Graylog log import problem

[bignjato (Boris Ignjatović)]

Hello,

I have problems with missing logs when importing logs from Graylog Collector Sidecar folder (NXLog import)

I was testing upload

log file with 30000 fields

Import test 1 - 29.967 logs imported
import test 2 - 22.939 logs imported
import test 3 - 27.646 logs imported

log file with 100 fields

Import test 1 - 100 logs imported
import test 2 - 100 logs imported
import test 3 - 100 logs imported

log file with 1000 fields

Import test 1 - 1000 logs imported
import test 2 - 1000 logs imported
import test 3 - 1000 logs imported

4 x log file with 100 fields

Import test 1 - 397 logs imported
import test 2 - 342 logs imported
import test 3 - 378 logs imported

Graylog ova with 6 CPU and 12 GB RAM
Configuration
Path: /var/opt/graylog/data/journal
Earliest entry:4 minutes ago
Maximum size:50.0GB
Maximum age:12 hours 0 minutes
Flush policy:Every 1,000,000 messages or 1 minutes 0 seconds

[jochen]

How exactly are you transporting the logs to Graylog?
How exactly are you verifying how many log messages have been ingested and indexed?

[bignjato (Boris Ignjatović)]

Hello @jochen

I import log files over Graylog Collector Sidecar folder (NXLog import) and then verify number of imported messages in search

every upload test I work on a clean base

Search by import file name

[jochen]

@bignjato

I import log files over Graylog Collector Sidecar folder (NXLog import)

What transport protocol are you using?

Raw/Plaintext? GELF? Syslog? Over TCP or UDP?

[bignjato (Boris Ignjatović)]

[jochen]

The suspense is killing me…

What type of input are you using in Graylog?

It seems to be UDP-based, so you should keep in mind that this is a stateless protocol (in contrast to TCP) and you might lose network packets without noticing.

[bignjato (Boris Ignjatović)]

Sorry for that,

I use graylog colector in the same machine and nxlog plugin to send logs from folder to graylog on the same machine.

Yes I use GELF UDP, what is your proposal to use TCP instead?

Sorry for my replays

[jochen]

@bignjato

what is your proposal to use TCP instead?

You could try that.

Also make sure to read this GitHub issue:

TCP Input messages "dropped" at high send volumes

[bignjato (Boris Ignjatović)]

Yes I try with tcp and 1 000 000 mesages goes ok I think that TCP will resolve all my problems!

Thanks you save me!

[bignjato (Boris Ignjatović)]

No error is the same when I import all my logs every time is different count of all logs

Import 1 - 17,951.131 messages
Import 2 - 9,671.136 messages
Import 3 - 16,245.694 messages

Before every test i clean indices and hotfolder.
No error on nxlog or collector-sideca.

What plugin you use for import manual logs with collector-sidecar??

this is setup for nxlog

define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>
<Extension 59949e3cca105203b1fb0d79-multiline>
  Module xm_multiline
  HeaderLine /^-./
</Extension>
<Extension 599bea29ca105203d800403c-multiline>
  Module xm_multiline
  HeaderLine /^\d{4}-\d{2}-\d{2}/
</Extension>

<Processor 59949e3cca105203b1fb0d79-buffer>
  Type Mem
  Module pm_buffer
  MaxSize 16384
</Processor>
<Processor 599bea29ca105203d800403c-buffer>
  Module pm_buffer
  MaxSize 16384
  Type Mem
</Processor>
<Processor 59a664bfca105208352055fc-buffer>
  Module pm_buffer
  MaxSize 16384
  Type Mem
</Processor>


User nxlog
Group nxlog

Moduledir /usr/lib/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid


define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>




<Input 59949e3cca105203b1fb0d79>
        Module im_file
        File '/swisslog/*.log*'
        PollInterval 120
        SavePos True
        ReadFromLast False
        Recursive True
        RenameCheck True
        Exec $FileName = file_name(); # Send file name with each message
        InputType 59949e3cca105203b1fb0d79-multiline
</Input>
<Input 599bea29ca105203d800403c>
        Module im_file
        File '/swisslog/*.txt*'
        PollInterval 30
        SavePos True
        ReadFromLast False
        Recursive True
        RenameCheck True
        Exec $FileName = file_name(); # Send file name with each message
        InputType 599bea29ca105203d800403c-multiline
</Input>
<Input 59a664bfca105208352055fc>
        Module im_file
        File '/swisslog/*.tsv*'
        PollInterval 5
        SavePos True
        ReadFromLast False
        Recursive True
        RenameCheck True
        Exec $FileName = file_name(); # Send file name with each message
</Input>





<Output 59949e20ca105203b1fb0d58>
        Module om_tcp
        Host 172.16.11.19
        Port 5045
        OutputType  GELF_TCP
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'f2fb322c-dcd3-44e8-b5c1-10fb0ba92188';
        Exec $collector_node_id = 'graylog-collector-sidecar';
        Exec $Hostname = hostname_fqdn();
</Output>
<Output 599bea29ca105203d800403b>
        Module om_tcp
        Host 172.16.11.19
        Port 5055
        OutputType  GELF_TCP
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'f2fb322c-dcd3-44e8-b5c1-10fb0ba92188';
        Exec $collector_node_id = 'graylog-collector-sidecar';
        Exec $Hostname = hostname_fqdn();
</Output>
<Output 59a664bfca105208352055fb>
        Module om_tcp
        Host 172.16.11.19
        Port 5060
        OutputType  GELF_TCP
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'f2fb322c-dcd3-44e8-b5c1-10fb0ba92188';
        Exec $collector_node_id = 'graylog-collector-sidecar';
        Exec $Hostname = hostname_fqdn();
</Output>


<Route route-0>
  Path 59949e3cca105203b1fb0d79 => 59949e3cca105203b1fb0d79-buffer => 59949e20ca105203b1fb0d58
</Route>
<Route route-1>
  Path 599bea29ca105203d800403c => 599bea29ca105203d800403c-buffer => 599bea29ca105203d800403b
</Route>
<Route route-2>
  Path 59a664bfca105208352055fc => 59a664bfca105208352055fc-buffer => 59a664bfca105208352055fb
</Route>