ERROR om_tcp detected a connection error; End of file found

OK, I am still trying to get our set up to work with Graylog and am having trouble with the sidecar-collector. Here is what I have:

server send logs from: Red Hat Enterprise Linux Server release 7.3, running graylog-collector-sidecar with nxlog. The collector is generating a nxlog.conf file like this:

define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>

User nxlog
Group nxlog

Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid

define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>

<Input 59fcda86ccba8e2573422cb4>
        Module im_file
        File '/var/log/httpd/*_access_log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>

<Output 59fcda56ccba8e2573422c80>
        Module om_tcp
        Host graylog.library.vanderbilt.edu
        Port 5044
        OutputType  GELF_TCP
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = '485f3ca7-ca1e-4959-be00-117a50e2b1db';
        Exec $collector_node_id = 'graylog-collector-sidecar';
        Exec $Hostname = hostname_fqdn();
</Output>

<Route route-0>
  Path 59fcda86ccba8e2573422cb4 => 59fcda56ccba8e2573422c80
</Route>

But in my nxlog.log on this server, I am seeing these messages:
2017-11-06 12:25:12 ERROR om_tcp detected a connection error;End of file found
2017-11-06 12:25:13 INFO connecting to graylog.library.vanderbilt.edu:5044
2017-11-06 12:25:24 INFO reconnecting in 1 seconds
2017-11-06 12:25:24 ERROR om_tcp detected a connection error;End of file found
2017-11-06 12:25:25 INFO connecting to graylog.library.vanderbilt.edu:5044
2017-11-06 12:26:08 INFO reconnecting in 1 seconds
2017-11-06 12:26:08 ERROR om_tcp detected a connection error;End of file found
2017-11-06 12:26:09 INFO connecting to graylog.library.vanderbilt.edu:5044

And on my Graylog server, I am seeing the following errors:
2017-11-06T12:28:50.670-06:00 ERROR [NettyTransport] Error in Input [Beats/59e8c322ccba8e11c8e9f514] (channel [id: 0x4ab85d41, /129.59.95.20:39170 => /129.59.151.61:5044])

So it appears my issue is with my INPUT setup on my Graylog server, but I do not know what to change or how to fix it. What I am trying to do is to send all of my Apache access log files to Graylog for analysis (and want to do this across all of our servers), but I just need help getting started. Please advise. Thank you!

Current INPUT configuration:
Name = apachein
Forward to (Required) = apacheout [nxlog]
Type = [NXLog] file input
Path to Logfile = /var/log/httpd/*_access_log
Poll Interval = 1
Save read position = checked
Read since start = checked
Recursive file lookup = checked
Rename check = unchecked
Enable multiline = unchecked
Start pattern of multiline = /^-./ (I am assuming that since “Enable multiline” is unchecked that this will be ignored)
Stop pattern of multiline =
Additional Fields =

Jamen McGranahan

Hej @jmcgranahan

is your input on Graylog a GELF TCP Input?

On my Graylog server, I do not have an option for GELF TCP Input. The options I have in my collector for Input are:
[NXLog] file input, Windows event log, UDP syslog listener, or TCP syslog listener. Currently I have it set to file input, pointing to my Output which does have a GELF TCP Output option (which I have selected).

Just to add more information: I have added the graylog-collector-sidecar to two other servers (one is a CentOS 7 box; the other another RedHat 7 box). The CentOS 7 box is on the same network as my Graylog server, but it is also giving the same error. So now I have 3 boxes trying to point to my Graylog server, with all 3 giving the exact same error:

2017-11-08 15:37:46 ERROR om_tcp detected a connection error;End of file found
2017-11-08 15:37:47 INFO connecting to 129.59.151.61:5044
2017-11-08 15:37:47 INFO reconnecting in 1 seconds
2017-11-08 15:37:47 ERROR om_tcp detected a connection error;Connection reset by peer
2017-11-08 15:37:47 INFO reconnecting in 2 seconds
2017-11-08 15:37:48 INFO connecting to 129.59.151.61:5044

let me clarify:

Did you have configured a Graylog Input ( can be found in System > Inputs in the Graylog Webinterface) that is of the type GELF TCP listening on Port 5044?

OK, I tried that but the Web Interface errored out - said the Input could not start, so I reverted it back to 12201. However, that made me think - what if I changed the port of the Collector from 5044 to 12201? Voila! That did it! Now I am getting my logs on my Graylog server! Thanks for the guidance!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.