Cannot perform search in UCS-2 encoded logs


(Loupasakis Kostas) #1

Hello everyone,

I am using graylog2 to manage the logs of a proprietary application we use in my team. The problem is that that application is writing it’s logs on files using UCS-2 LE BOM encoding.

We use the Graylog Collector Sidecar in order to parse our logs.

All the logs reach our graylog2 server without problem, but possibly because of the obscure encoding all the searches we do return no results.
Similarly I try to create Regex extractors for the same logs, but patterns that should obviously match don’t.

Does anyone know of a way we could transform the logs to UTF-8 before we reach graylog2? Unfortunately we cannot change the encoding on the level of the application.

Thank you :slight_smile:


(Jochen) #2

What type of logs are you parsing, what actual collector are you using, and what’s its specific configuration?


(Chrysanthos Kattimeris) #3

@jochen
thanks a lot for the input to our issue
we are parsing log: ASCII text, with CRLF line terminators

we can’t change it to UTF8…
as per configuration we have the following beats input :http://take.ms/SRz9o


(Jochen) #4

NXLOG supports converting between character sets: https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#xm_charconv


(Chrysanthos Kattimeris) #5

thanks a lot for the time to help us

please note i’ve created the following on our graylog : http://take.ms/eWyKo

our problme is that is not reading the file in realtime it has a delay of 5-6 minutes

below is the configuration of our nxlog

define ROOT C:\Program Files (x86)\nxlog

Module xm_gelf

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

Module xm_fileop When @daily Exec file_cycle('%ROOT%\data\nxlog.log', 7);

<Input 599ea1aee3ae2946581c20d0>
Module im_file
File 'D:\MetaTrader4Server\logs\2*.log’
PollInterval 1
SavePos True
ReadFromLast True
Recursive True
RenameCheck False
Exec $FileName = file_name(); # Send file name with each message
Exec convert_fields(“AUTO”, “utf-8”);

<Output 599ea17ee3ae2946581c209d>
Module om_tcp
Host vip.tradingpoint.com
Port 12201
OutputType GELF_TCP
Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
Exec $gl2_source_collector = ‘f6cee18b-00a1-4f46-b35a-4b85d66546c3’;
Exec $collector_node_id = ‘graylog-collector-sidecar’;
Exec $Hostname = hostname_fqdn();

Path 599ea1aee3ae2946581c20d0 => 599ea17ee3ae2946581c209d

(Jochen) #6

Are you sure about that? Try comparing the timestamps in your log files and the timestamps displayed in Graylog. Also make sure that the system clocks are in sync (e. g. by using NTP).

Please also make sure to format your text snippets properly: http://commonmark.org/help/
Example:

```
TEXT
```

(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.