Graylog - Sidecar question

I have been using Graylog with a locally installed sidecar - to ingest logfiles taken from an server that is no longer in use.
The files were ingested as raw messages, however I have since found out that I can process the file as a CSV (decode_csv).

I wanted to clear the data this sidecar has harvested,
stopped Filebeat,
removed the files in /var/lib/graylog-sidecar/collectors/filebeat/
rotated and removed old indexes

The messages are no longer visible in graylog but when I access system / sidecars
and click on my sidecar to view the status, the screen Im shown takes ages to load - the files are still referenced.

How do I clear this and how do I reduce list of files shown?
Mark

Hello,

Navigate to System/sidecar/Adminstator highlight then Clear selection.

You will be greeted with the following

Should be good, Might have to refresh your browser.

Hope that helps

Thanks for your reply,
I managed to work out what was wrong.
In my /etc/graylog/sidecar/sidecar.yml file I had uncommented list_log_files and put some entries in between the square brackets.

#
# Default: empty list
#list_log_files: []


I am new to Graylog and this was where I thought you tell the sidecar to harvest logs from - i know now that this was wrong, I had forgotten that I had made this change and once I removed it, the lists have gone.

I thought everything was working until i spotted that the dates from the raw messages are being parsed.

my raw messages

Interim-Update,user5@realm,2021/09/16 14:42:32,0,HOST-1,3174790000001AA1431119,10.10.110.112,Framed-User,PPP,,,10.20.126.116,255.255.255.255,,,,122800,,124584,,1350,,1344,PPPoEoQinQ,180056165,lag-10:999.101,Radius,

all messages are pushed though “decode_csv_fields”
e.g.

processors:
  - decode_csv_fields:
      fields:
        message: csv
      overwrite_keys: true

  - extract_array:
      field: csv
      overwrite_keys: true
      omit_empty: true
      mappings:
        Acct-Status-Type: 0
        User-Name: 1
        Event_Timestamp: 2
        Acct-Delay-Time: 3
        NAS-Identifier: 4
        Acct-Session-Id: 5
        NAS-IP-Address: 6
        Service-Type: 7
        Framed-Protocol: 8
        Framed-Compression: 9
        Unisphere-PPPoE-Description: 10
        Framed-IP-Address: 11
        Framed-IP-Netmask: 12
        Unisphere-Ingress-Policy-Name: 13
        Calling-Station-Id: 14
        Acct-Input-Gigawords: 15
        Acct-Input-Octets: 16
        Acct-Output-Gigawords: 17
        Acct-Output-Octets: 18
        Unisphere-Input-Gigapackets: 19
        Acct-Input-Packets: 20
        Unisphere-Output-Gigapackets: 21
        Acct-Output-Packets: 22
        NAS-Port-Type: 23
        NAS-Port: 24
        NAS-Port-Id: 25
        Acct-Authentic: 26
        Acct-Session-Time: 27
        Delegated-IPv6-Prefix: 28

and these are split nicely in to the fields that I can search

what I was not expecting is that the field Event_Timestamp is being parsed and instead of showing as

2021-09-16 14:42:32.000 +01:00

its

2021-09-16 15:42:32.000 +01:00

how can I prevent this and at the same time, how can I overwrite timestamp with this field value?
thanks

Hello,
This is kind of a broad question. what I would do is check the following first.

  • Does the device sending logs have the correct Date/Time.
  • Graylog server have the correct time (NTP) and configuration file root_timezone = is correct.
  • Insure that the user logged in has the correct Time settings.

  • check under System/Overview in the Web UI, these should be the same.

EDIT:

My apologies, I missed read your statement. I though you want to remove the collector configuration.

Hope that helps.

So this has me puzzled -

I have tried this with and without a timezone set in my profile
with I use isle_of_man which is basically London BST (GMT+1)

Inkedtime_LI

If I view any entry from either as

/messages/graylog_7/a8758d31-1788-11ec-8577-ac162d8befa8
or
/api/messages/graylog_7/a8758d31-1788-11ec-8577-ac162d8befa8

the value I see presented in the Event_timestamps as it should be

e.g.

yet the same entry when viewed from the default dashboard, the time that is shown for Event_timestamp is 1hour ahead…

and what is even more confusing, is that the @ timestamp field is correct when viewing it from the default dashboard but and hour behind when viewing just that message.

is this a bug?

Not that I Know of. Elasticsearch stores all “time” fields in UTC/Epoch. I would double check your times/Date on each device and to make sure there not a problem.

TimeStamp 2021-09-17 08:27;05.496
@timestamp 2021-09-17T07:27:04.088Z
Event_Timestamp 2021/09/17/ 08:26:54

You have something crazy going on, not sure what.

Maybe some of these post will help

https://community.graylog.org/search?q=Timestamp

EDIT: I read over this post trying to figure out an awswer for you about the hour difference.
Just realize something. If Elasticsearch stores all “time” fields in UTC/Epoch ‎and UTC · ‎Greenwich Mean Time (GMT) . Then your statement was

There is your hour difference. I’m not 100% sure but a pipeline might be in your future to set these timestamps correct.