Sidecar Input ignores encoding types?

[Dave_Hill]

I’m at a complete loss for reading some log files with FIX formatting. The sidecar I’ve created is ingesting the logs but is inserting unicode characters. I’ve tried UTF8, ANSII, WIndows encoding, and they all are producing the same characters in graylog. If I cat the file in linux, it reads the file fine, so i thought maybe ANSII was the key, but that one even creates the unicode characters

filebeat_agent_version: 7.8.0

The file -bi output looks like this:
application/octet-stream; charset=binary

# Needed for Graylog
fields.source: ${sidecar.nodeName}
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
logging.level: debug

escape_html: false

filebeat.inputs:
- type: log
  paths:
- /opt/XXX/XXX/logs/uat/W*Log*.LOG
  encoding: "UTF8"
  ignore_older: 48h
  
output.logstash:
   hosts: ["monitor.rXXXXs.com:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

[shoothub]

1. Graylog is capable only process UTF-8 messages, so please provide some example messages from log file…

2. Try to use e.g. program uchardet to detect encoding of file.

[Dave_Hill]

uchardet outputs ASCII, which graylog also creates unicode characters with. The FIX character that it uses is \x01 between each tag, if that helps at all.

20200709-21:59:35.000  8=FIX.4.49=0008335=049=MatrixFIXUAT56=XXXXXXXX34=000126052=XXXXXX-21:59:35.00050=XXXXXX3310=151
20200709-22:00:00.042  35=549=XXXXXXX56=MatrixFIXUAT34=126152=XXXXX-22:00:00.00910=238