Various basic questions - Log Truncation, Formats, time stamps, etc

Graylog Central (peer support)
1

Hi,
I’m attempting to solve a few inherent issues with Unraid, all revolving around logs, and pushing them into Graylog.

As a pre-req, I am new to Graylog, I have read as much of the docs as I can, however without knowing what does what, its a bit of a needle in a haystack. So obvious answers to experienced Graylog’ers such as ‘well just use X, its in the docs’, is not going to be obvious to me.
So kid gloves would be appreciated, I’m here to learn. :slight_smile:

Problem Area 1: Unraid
Unraid stores its logs in RAM, so should the system reboot or lock up…you lose the logs.
There’s some creative ways about it, like having a cron job run to flush the logs to another device on a schedule, but I’d rather start telling all my devices and services to send logs to one location (Graylog), running on a self-contained system. In this case, a Raspberry Pi.

Context for issue
The first issue is that I’m finding that Unraid (and UniFi in all fairness) is sending the syslog message with incorrect timestamps (4 hours in the past), which when ingested into Graylog means that ‘recent queries’, or anything that relies on them, are next to useless.
The timezone is set correctly at both ends.

The “workaround” is to flip the input type from ‘Syslog/UDP’ to ‘RAW UDP’, which is not ideal, as you lose some of the automatic parsing being done.
But it does mean that the message timestamps in use are correct and usable, if if the timestamp in the text itself remains wrong.

Why do I know Graylog is behaving?
I’ve added the relevant extra parameters to my containers to push logs with tags into Graylog.
These all come into Graylog with everything correct.
My logic is that if Graylog had an inherrant config issue, it would affect that too…?

Q1
Is there a way that I can solve this Graylog side?
Is there a way to ‘overwrite’ the incoming timestamp perhaps?
Or automatically truncate the first 20 characters (that contain the time stamps) perhaps?
Note: I’d still want to ‘solve’ this on Graylog’s side irrespective of Unraid, as UniFi has the same issue.

Q2
Is this likely to be a rsyslog service config error in Unraid?
As if so, I would rather fix at source instead of workaround it (in the case of Unraid).
Could it be a format issue?

Problem Area 2: General Usage Questions
Q3: I’ve been splitting my incoming inputs into their own streams, then filtering into their own indicies to help seperate them from a data perspective. Is this good, bad or pointless to do?

Q4 Plugins and content packs - limited reading thus far, but surely i do not need to create custom docker images just to add a plugin? Is it not just a case of drop into a directory, restart the container and go?

Q5 Any other general tips?

Thank you in advance for your help with these probably basic questions. :slight_smile:

2

Hello && Welcome @boomam

I’m going to try to answer your questions :+1:

This question will have a few answers or suggestions, its kind of broad, but lets take the 4 hours in the past issue.
This may have to do with NTP, either on your client device/s or logging server. When this occurs I check the User profile settings for the correct Time Zone, The Virtual machine or physical server is correct , and also the Graylog Configuration file.
As stated earlier the Client/s also must be synced to the same time zone.
If this is all corrected and the issue still does occur then a pipeline or most extractor will be need to correct this.
I probably could name at least 4 ways to correct the time Zone on log files this all depends on which way you want to go and resources you have.
Some configurations will take more resources then others.

I think I may have answered this question above. If not explain in greater detail.

This could help and I also do this. Side note I click the tick box to remove messages from " All messages" stream so there are not copies of the same message indexed.

This depends on how the set up of Graylog Docker. I personally use Docker-Compose set it up so its easy, a drop and restart situation. But content packs you can upload those through e Web UI.

Example:

environment:
  - graylog_plugin:/usr/share/graylog/data/plugin

Dont take candy from a stranger :slight_smile: But seriously if you want tips and learn more hangout here and ask question or go over to Graylog Discord channel.

3

Good morning @gsmith ,

Thank you for the reply!

RE: Q1/NTP
I can confirm that all my devices/time settings are in sync & correct - I use a private NTP on my local LAN to help with that.

RE: Q1/4 different ways to correct it.
Could you expand/link to the methods that come to mind? I can then do my own research and probably loop back should I have issues or queries on them.

RE: Q3/“Remove messages” box.
Yup, ticked that. Didnt see a reason to duplicate data. :slight_smile:

RE: Q4/Plugins
Ah, did not see that upload button before. Well thats much easier! :slight_smile:
I’ll check the path you reference is covered in one of the persistant volumes, first, though.

RE:Q5/Discord
I’ll have to see if i can find the link, a Discord could be handy to ask random/quick questions & contribute more actively too.

Thanks!

4

Hey @boomam

How about user/s time/date?

image
image1129×555 44.9 KB

Here is a global search for Timestamp, this may help. Warning there are a lot :slight_smile:

Example of a pipeline, this is one of many and it depends on your situation.

rule "Epoch Convert"
when
  has_field("eventtime")
then
  let ts_millis = to_long($message.eventtime) / 1000;
  let new_date = parse_unix_milliseconds(ts_millis);
  set_field("epoch_timestamp", new_date); 
end

Regex Extractor’s, there is a drop down at the bottom of the page where is says “Add converter”… click on Date.

image
image1282×780 48.2 KB

5

Hi @gsmith ,

RE: Time
Users times are correct, however I recently found out that all admins list the timestamps as UTC regardless of that setting. Is that correct?

As if so, based on that, of the sources I have, doing a quick test when logged in as one the admins I have setup -

Actual Time: 6:29-6:35pm (US EST)
System Time: 6:29-6:35pm (US EST)

Input Type Time at time of test Time on message Timestamp Verdict
Raspberry Pi Syslog UDP 18:29 22:29 22:29 +4 hours, equaling UTC
UniFi RAW/Plaintext UDP 18:35 22:35 22:35 +4 hours, equaling UTC
Unraid Syslog Syslog UDP 18:32 18:32 18:32 Exactly the same time
Unraid Containers (Docker) Syslog UDP 18:34 22:34 22:34 +4 hours, equaling UTC

The only anomaly there is Unraid’s Syslog. Maybe the time is getting overridden in some way?

If the above is ‘correct’ then the methods you mention, may not even be needed anymore…

Lastly, thanks for the Discord link! :slight_smile:

Thanks.

1 Like
6

Hello,

I’m not sure about UniFi or Pi Input but the Unraid Containers (Docker).
I had to use the following.

environment:
      # Container time Zone
      - TZ=America/Chicago

What I have seen in the past with the “timestamp” on the messages diff from the actual time was the device or the log shipper was incorrect. As stated above, you clarified that all devices time/date were correct. Not sure if this was tried but the actual log on the remote device check the timestamp if its correct then, I have to agree, something is changing it before it hits Graylog.

7

Looking at my docker-compose, i have this set -

   - GRAYLOG_ROOT_TIMEZONE=UTC

I think i set it to that as i read its better to leave it as UTC and just rely on the users preferences instead (which of course, if im correct, dont apply to admin users).
Does the existing data get auto-translated to whatever timezone i change that too on next boot? Or just new data?

8

Hello,
Here is mine, graylog root time and Container time is different The UTC from EST time is your 4 hour difference I do believe.

 environment:
      # Container time Zone
      - TZ=America/Chicago

      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=pJod1TRZAckHmqM2oQPqX1qnLZuWgMtnw2nonDhN

      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=ef92b778bafe771e8921881f383d4473e94f
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_EXTERNAL_URI=http://ip_address:9000/
      - GRAYLOG_ROOT_TIMEZONE=America/Chicago
      - GRAYLOG_ROOT_EMAIL=greg.smith@domain.com
      - GRAYLOG_HTTP_PUBLISH_URI=http://ip_address:9000/
      - GRAYLOG_TRANSPORT_EMAIL_PROTOCOL=smtp
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL=http://ip_address:9000/
      - GRAYLOG_TRANSPORT_EMAIL_HOSTNAME=ip _address
      - GRAYLOG_TRANSPORT_EMAIL_ENABLED=true
      - GRAYLOG_TRANSPORT_EMAIL_PORT=25
      - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
      - GRAYLOG_TRANSPORT_FROM_EMAIL=root@localhost
      - GRAYLOG_TRANSPORT_SUBJECT_PREFIX=[graylog]      
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:

EDIT: On Graylog Web UI under System/Overview you should see this. BTW this is my Docker installation.

image
image885×416 26 KB

9

Sorry I missed this statement,

I believe its the new data but not 100% sure, If you do correct the time zone, I would manually rotate the index set. or wait for the next rotation that should get things in order.

10

ok, if I’ve understood correctly, I’ll set both the TZ and the Graylog variable to EST?

To rotate the index, I dont see a function to do that.
Nearest I can find is “Rotate Active Write Index” ?

Once this is sorted (managed to fix a few other things in the background), i can move onto tuning out the white noise, as a pipeline rule appears to work in a simulator, but not in in actuality…

11

Doesnt hurt to try, its just time :wink: BTW mine it set to central time zone.
So…

The time zone setting of the root user. See this list of valid time zones. Default is UTC.

That is correct. Depending on how many logs are coming it it might take a few minutes.

12

Hey @boomam

Just so you know I’m not making this up, here are some ref in the forum I found for ya.

13

Thanks for the links. :slight_smile: Good reading.
Timezone changes done, I’ll test for a day or so, see how it shakes out.

One thing i did notice post-timezone changes, Indexer failures.
Did a bit of a search, had to do some mongo collection drops to wipe the errors (which seems an odd way to achieve it…).

14

Hello @boomam

Good to hear :+1:

I would need to see the whole error log to make a suggestion.

15

Oh, its just the ‘normal’ status message in system->overview.
Once cleared out, i had a few more return, but nothing since. So I’m guessing something to do with the time change.

Things seem to be behaving now though, which is great.
Or at least enough for me to start writing up/documenting it for myself on my doc site at least. :-p

Just the rule sets to drop some messages to sort that i can think of now. :slight_smile:

16

Oh i see, well hope things work out for you.
If things are working for ya don’t forget to mark this post as resolved for future search, Im quite sure someone later will have the same problem :+1:

Just a reminder if you need to ask a quick question head over to Discord. I’m not in the forum on weekends. Most, if not all of us are playing video games :smiley:

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.