Messy code in messages

Graylog Central (peer support)
, , , ,
1

got messages ware be messy code,when i try to collect log from /var/log/httpd/access_log (apache access log) by collector_sidecar.
i don’t know why…how can i resolve it?

微信截图_20170613231033.png1323×412 73.5 KB

2

Which log shipper are you using, NXLOG or Filebeat?
How did you configure the Graylog Collector Sidecar?
How did you configure the log shipper (i. e. what’s the generated configuration)?
Which locale are you using on the system with the Apache httpd access logs?

3

Thank you for replay…
>Which log shipper are you using, NXLOG or Filebeat?
NGLOG

>How did you configure the Graylog Collector Sidecar?
the configure of Collector Sidecar as below:

server_url: http://192.168.2.9:12900
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
    - /var/log/httpd/
node_id: IAMANODE
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
    - apache
backends:
    - name: nxlog
      enabled: true
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/nxlog.conf

>How did you configure the log shipper (i. e. what’s the generated configuration)?
i don’t know what’s the mean of log shipper , means INPUTS?
if INPUTS,i defined it like below:

image.png1330×919 86.8 KB

>Which locale are you using on the system with the Apache httpd access logs?
collector-sidecar OS:CentOS7.1

[root@cmdb ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

access_log is us-ascii, as below

[root@cmdb ~]# file -i /var/log/httpd/access_log
/var/log/httpd/access_log: text/plain; charset=us-ascii
4

and this is my snippet config:

{{if .Linux}}
User nxlog
Group nxlog
{{if eq .LinuxPlatform "debian"}}
Moduledir /usr/lib/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid
{{end}}
{{if eq .LinuxPlatform "redhat"}}
Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid
{{end}}
define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>
<Extension charconv>
  Module xm_charconv
  AutodetectCharsets AUTO,utf-8
</Extension>
{{end}}
{{if .Windows}}
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>
{{end}}

also added verbatim configuration in the end of INPUT:

Exec convert_fields("AUTO", "utf-8");
5

Please post the contents of /etc/nxlog.conf.

6

below is my nxlog.conf:

define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>

<Processor 5940a8982e88b604b73f72fb-buffer>
  Module pm_buffer
  MaxSize 16384
  Type Mem
</Processor>

User nxlog
Group nxlog

Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid

define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>
<Extension charconv>
  Module xm_charconv
  AutodetectCharsets AUTO,utf-8
</Extension>

<Input 5940a8982e88b604b73f72fb>
	Module im_file
	File '/var/log/cmdbd.log'
	PollInterval 1
	SavePos	True
	ReadFromLast True
	Recursive True
	RenameCheck False
	Exec $FileName = file_name(); # Send file name with each message
	Exec convert_fields("AUTO", "utf-8");
</Input>




<Output 593fb8bf2e88b6055d1b5de6>
	Module om_udp
	Host 192.168.2.9
	Port 1514
	OutputType  GELF
	Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
	Exec $gl2_source_collector = 'aa44b4ff-7f75-4c89-96e1-7cedd6a40945';
	Exec $collector_node_id = 'IAMANODE';
</Output>

<Route route-0>
  Path 5940a8982e88b604b73f72fb => 5940a8982e88b604b73f72fb-buffer => 593fb8bf2e88b6055d1b5de6
</Route>
7 
define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>

<Processor 5940a8982e88b604b73f72fb-buffer>
  Module pm_buffer
  MaxSize 16384
  Type Mem
</Processor>


User nxlog
Group nxlog


Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid

define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>
<Extension charconv>
  Module xm_charconv
  AutodetectCharsets AUTO,utf-8
</Extension>




<Input 5940a8982e88b604b73f72fb>
	Module im_file
	File '/var/log/cmdbd.log'
	PollInterval 1
	SavePos	True
	ReadFromLast True
	Recursive True
	RenameCheck False
	Exec $FileName = file_name(); # Send file name with each message
	Exec convert_fields("AUTO", "utf-8");
</Input>




<Output 593fb8bf2e88b6055d1b5de6>
	Module om_udp
	Host 192.168.2.9
	Port 1514
	OutputType  GELF
	Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
	Exec $gl2_source_collector = 'aa44b4ff-7f75-4c89-96e1-7cedd6a40945';
	Exec $collector_node_id = 'IAMANODE';
</Output>



<Route route-0>
  Path 5940a8982e88b604b73f72fb => 5940a8982e88b604b73f72fb-buffer => 593fb8bf2e88b6055d1b5de6
</Route>
8

image.png857×872 19.1 KB

9

i try to command the OutputType
and got right message
maybe reason is input message’s format is not mismatching to GELF??

image.png869×240 6.16 KB

10

this issue has been resolved.
reason is collector_sidecar output into the wrong INPUT format(syslog type).
create a new type INPUT(GELF) and change collector_sidecar output to the new INPUT,everything is OK!

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.