Send ".log" logs to graylog server 4.2

mxlvs · January 6, 2023, 2:28pm

Hi,

I have a filezilla server installed on a windows machine that generates logs in (C:\Program Files (x86)\Filezilla Server\Logs*.log) and I needed to send these logs to my graylog server. I couldn’t find a way to send it via nxlog.

Debian 11 - Graylog version 4.2.12-1 - 192.168.1.50
Windows 10 - Nxlog version 3.1.2319 - 192.168.1.51

My nxlog.conf configuration

Panic Soft
 #NoFreeOnExit TRUE
 
 define ROOT     C:\Program Files\nxlog
 define CERTDIR  %ROOT%\cert
 define CONFDIR  %ROOT%\conf
 define LOGDIR   %ROOT%\data
 define LOGFILE  %LOGDIR%\nxlog.log
 LogFile %LOGFILE%
 
 Moduledir %ROOT%\modules
 CacheDir  %ROOT%\data
 Pidfile   %ROOT%\data\nxlog.pid
 SpoolDir  %ROOT%\data
 
 <Extension _syslog>
     Module      xm_syslog
 </Extension>
 
 <Extension _charconv>
     Module      xm_charconv
     AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
 
<Extension _exec>
     Module      xm_exec
</Extension>
 
<Extension gelf>
     Module      xm_gelf
</Extension>
 
<Extension _fileop>
#     Module      xm_fileop
 
     # Check the size of our log file hourly, rotate if larger than 5MB
     <Schedule>
         Every   1 hour
         Exec    if (file_exists('%LOGFILE%') and \
                    (file_size('%LOGFILE%') >= 5M)) \
                     file_cycle('%LOGFILE%', 2);
     </Schedule>
 
     # Rotate our log file every week on Sunday at midnight
     <Schedule>
         When    @weekly
         Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
     </Schedule>
 </Extension>
 
 <Input in>
     Module      im_msvistalog
 </Input>
 
 #<Input mvfiles>
 #   Module      im_file
 #	SavePos		FALSE
 #	#Recursive	TRUE
 #	ReadFromLast	FALSE
 #	InputType	multiline
 #	Exec parse_syslog();
 #	PollInterval 10
 #	SavePos False
 #	ReadFromLast True
 #	Recursive True
 #	RenameCheck False
 #Exec $FileName = filename();
 #</Input>
 
 <Output out>
     Module      om_tcp
     Host        IP
     Port        12202
     OutputType  GELF_TCP
 #Exec $shor_message = $raw_event;
 #Exec $gl12_source_colletor =  9ee4ff87-aa25-486d-9980-7a0de8328980';
 #Exec $colletor_node_id = 'DOMAIN';
 #Exec $Hostname = hostname_fqdn();
 </Output>
 <Route 1>
     Path in => out
 </Route>

would like to know if it is possible to send the contents of these *.log files from my filezilla server via nxlog or another software to my graylog server and that I can handle this information there.

Thank you very much for the support

tmacgbay · January 6, 2023, 2:32pm

You could use NXlog (@gsmith is better at that than me) or you could use Filebeat… which maybe be included with the winlogbeat on the Sidecar install… if not you can download it from Elastic easily. If you Stick with NXlog, @gsmith can give you a generic configuration (or you can search for one in the forums) if you change to filebeat, I can post a generic configuration.

mxlvs · January 6, 2023, 2:41pm

@tmacgbay thank you very much for the quick response.
I’ve never worked with filebeat and I don’t even know where to start. but if it is an option, I can migrate without problems.

would it be possible for me to install filebeat on my windows 10 machine to send these “.logs” and keep nxlog sending the windows logs normally, using both solutions? or should i center it only on the filebeat?

mxlvs · January 6, 2023, 3:24pm

@gsmith

Can you help me and share with me an nxlog configuration template so I can upload my .log files?

thank you very much in advance.

tmacgbay · January 6, 2023, 5:55pm

If you choose to change to beats, I am always a fan of consistency. @gsmith is usually busy during the day but might be able to get back to you this evening. If you go with beats, here is a (Somewhat Obfuscated) configuration I have for some Windows machines:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["${gl.receive}"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security
     processors:
       - drop_event.when:
           or:
             - equals.winlog.event_id: "####"
       - drop_event.when:
           and:
             - equals.winlog.event_id: "####"
             - regexp.winlog.event_data.SubjectUserName: '^Catw'
       - drop_event.when:
           and:
             - equals.winlog.event_id: "####"
             - equals.winlog.event_data.SubjectUserName: "Robin"
       - drop_event.when:
           and:
             - equals.winlog.event_id: "####"
             - equals.winlog.event_data.TargetUserName: "Batman"

# Pickup on print jobs but not the rest of the malarky - we only want event 307
   - name: Microsoft-Windows-PrintService/Operational
     processors:
       - drop_event.when:
           or:
             - equals.winlog.event_id: "801"
             - equals.winlog.event_id: "800"
             - equals.winlog.event_id: "805"
             - equals.winlog.event_id: "842"

Here is one for windows filebeat:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
   hosts: 
   - ${ugl.receive}
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - win_file
processors:
  - add_fields:
      target: ''
      fields:
        field_a: AAAA
        field_b: BBBBB
filebeat:
  inputs:
    - type: log
      paths:
        - C:\Program Files\Graylog\testing.log

mxlvs · January 6, 2023, 6:04pm

Nice @tmacgbay
I really appreciate all the information you sent me.

I didn’t know about filebeat and I can’t succeed with nxlog, I’ll study more about it to try to load the graylog for it.

tmacgbay · January 6, 2023, 6:07pm

Forgot to post Graylogs Beats page in docs: Ingest from files

gsmith · January 6, 2023, 10:14pm

hey @mxlvs

Sure, , you will need to make adjustments to this config for your environement thou.

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log


Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
#LogLevel DEBUG

<Extension _fileop>
    Module xm_fileop
    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>
    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

<Extension gelf>
    Module      xm_gelf
 </Extension>
<Input zone-01>
    Module      im_msvistalog
    Query <QueryList>\
     <Query Id="0">\
      <Select Path="Application">*</Select>\
      <Select Path="System">*</Select>\
      <Select Path="Security">*</Select>\
      <Select Path="Setup">*</Select>\
     </Query>\
    </QueryList>  
</Input>

<Output out>
    Module      om_ssl 
    Host        graylog.domain.com
    Port        51412
    OutputType  GELF_TCP 
    CertFile    %CERTDIR%/graylog3-certificate.pem
    CertKeyFile %CERTDIR%/graylog3-key.pem
    CAFile      %CERTDIR%/cert3.pem
    KeyPass     secret 
    AllowUntrusted  true   
    Exec $Hostname = hostname_fqdn();
    Exec $FullMessage = $raw_event;
    #Exec        to_syslog_snare();
</Output>

<Route >
    Path        zone-01 => out
</Route>

mxlvs · January 10, 2023, 1:43pm

thank you very much for the feedback. @gsmith

I haven’t had time to test and validate yet, but I’ll get back to you soon.

mxlvs · January 11, 2023, 1:55pm

I apologize for the delay, I was busy with other requests.

@gsmith
A doubt, knowing that my filezilla server is recording the logs in “C:\Program Files (x86)\Filezilla Server\Logs\Filezilla Server.log”. how is the parameterization of my nxlog.conf file?

I tried this way, however, I already received the error 1053 for running the service.

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log
define FTP C:\Program Files (x86)\FileZilla Server\Logs\FileZilla Server.log

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogFile %FTP%\FileZilla Server.log

#LogLevel DEBUG

<Extension _fileop>
    Module xm_fileop
    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>
    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

<Extension gelf>
    Module      xm_gelf
 </Extension>
<Input zone-01>
    Module      im_msvistalog
    Query <QueryList>\
     <Query Id="0">\
      <Select Path="Application">*</Select>\
      <Select Path="System">*</Select>\
      <Select Path="Security">*</Select>\
      <Select Path="Setup">*</Select>\
     </Query>\
    </QueryList>  
</Input>

<Output out>
    Module      om_ssl 
    Host        graylog.domain.com
    Port        51412
    OutputType  GELF_TCP 
    CertFile    %CERTDIR%/graylog3-certificate.pem
    CertKeyFile %CERTDIR%/graylog3-key.pem
    CAFile      %CERTDIR%/cert3.pem
    KeyPass     secret 
    AllowUntrusted  true   
    Exec $Hostname = hostname_fqdn();
    Exec $FullMessage = $raw_event;
    #Exec        to_syslog_snare();
</Output>

<Route >
    Path        zone-01 => out
</Route>

Thanks in advance.

gsmith · January 11, 2023, 10:35pm

Hey @mxlvs

If you triing to use my nxlog template it will not work for you. It was configured for my environment.

Not sure what type of file FileZilla log file is you may or may not need to add an Extension .

Next, create a input to get that file.

<Input filezilla>
    Module       im_file
    FILE         "C:\Program Files (x86)\Filezilla Server\Logs\Filezilla Server.log"
    SavePos       TRUE
    ReadFromLast  TRUE
    PollInterval  1
    Exec  $Message = $raw_event;
</Input>

Then add your route. This would tie the INPUT and OUtput being used.

<Route >
    Path      filezilla <name of input> => out <name of output>
</Route>

Your output configuration should be setup for the Type of INPUT your send logs to Graylog.
Let’s say Syslog UDP, then it would look somehting like this.

<Output out>
    Module      om_udp
    Host        graylog.domain.com
    Port        5140
    Exec  $Message = $raw_event;
</Output>

So all together you would have something this.

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

### This is for Nxlog syslog file. 

LogFile %ROOT%\data\nxlog.log
#LogLevel DEBUG <---Only enable Debug mode to resolve issue the  comment it back out. this will create a large file over time.


### This is for rotating Nxlog log file.
<Extension _fileop>
    Module xm_fileop
    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>
    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

### Extentions for file types.

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension _json>
    Module      xm_json
</Extension>

<Extension gelf>
    Module      xm_gelf
 </Extension>
 
 <Input filezilla>
    Module       im_file
    FILE         "C:\Program Files (x86)\Filezilla Server\Logs\Filezilla Server.log"
    SavePos       TRUE
    ReadFromLast  TRUE
    PollInterval  1    
</Input>

<Input windows_event>
    Module      im_msvistalog    
</Input>

<Output out>
    Module      om_udp
    Host        8.8.8.8 <---  Put you Graylog address here (FQDN or IP address)
    Port        5140  <-- port that would match Graylog INPUT  
    Exec  $Message = $raw_event;	
</Output>

<Route >
    Path        windows_event => out
</Route>
<Route >
    Path        filezilla => out
</Route>

Hope that helps

tmacgbay · January 12, 2023, 4:01pm

…That port might be having an identity crisis…

gsmith · January 12, 2023, 5:13pm

Thx @tmacgbay good catch,

mxlvs · January 23, 2023, 3:27pm

hello friends, sorry for the long period to return.
@gsmith and @tmacgbay I managed to forward my logs with .conf that you sent me. great! you are awesome!!!

however, for filezilla rotate files (ex: fzs-2023-01-23.log) it does not recognize and I have tried the following ways:

C:\Program Files (x86)\Filezilla Server\Logs\fzs-.log
C:\Program Files (x86)\Filezilla Server\Logs\fzs-

no way I managed to send the logs

what am I doing wrong?

gsmith · January 23, 2023, 11:41pm

Hey,

Maybe try this

C:\Program Files (x86)\Filezilla Server\Logs\*.log

i read the doc on filezilla and notice ther enot a way to just create a log file without the timestamp on it.

system · February 6, 2023, 11:42pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
NxLog Help - Logs Not Fully Ingesting Graylog Central (peer support) nxlog	4	255	July 14, 2023
61 / 5000 Resultados de traducción I need help sending IIS events to graylog with NXLog Graylog Central (peer support) sidecar , nxlog , nodatanx , send-csv-logsnx	2	661	August 17, 2021
Unable to send csv logs to graylog udp Graylog Central (peer support) sidecar , nxlog , nodatanx , send-csv-logsnx	1	1023	September 13, 2018
Setup of Graylog and NXLog Graylog Central (peer support) nxlog	14	2681	January 27, 2023
NXLog to Gray Log Graylog Central (peer support) sidecar , nxlog , nodatanx	6	1095	February 4, 2020

Send ".log" logs to graylog server 4.2

Related Topics