2.4.6 Log Rotation

Graylog Central (peer support)
1

Hi there,
could anyone help me configure log rotation?
I don’t have the graylog-ctl scripts for some reason so if it’s a part of that package then maybe I’m SOL.

Anyways,
I checked the server.conf and made some changes to the indexing but as far as I understand that’s for the purpose of being able to quickly search through the logs later. I also tried to configure logrotate, but when I couldn’t find where the old logs were being moved to I asked online & was told it’s because Graylog has its own rotation that it uses. I’ve seen the /etc/graylog/server/log4j2.xml file, but it isn’t immediately obvious to me how to configure it for log rotation.

Any help appreciated,

Thanks

2

Depending how you installed Graylog the location of the log configuration is a little different. Please check the Documentation about the location: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

You need to check your log4j configuration and might modify this if the defaults are not what you like to have.
Some help you might find on this page: https://www.codejava.net/coding/configure-log4j-for-creating-daily-rolling-log-files

1 Like
3

Hi Jan,
Thanks for the reply.

I wanted to make my log rotate at 2GB in size and keep 5 rotations.
I tried this:

<?xml version="0.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
<Appenders>
    <RollingFile name="rolling-file" fileName="/var/log/graylog-server/server.log" filePattern="/var/log/graylog-server/server.log.%i.gz">
        <PatternLayout pattern="%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%c{1}] %m%n"/>
        <Policies>
            <SizeBasedTriggeringPolicy size="2GB"/>
        </Policies>
        <DefaultRolloverStrategy max="5" fileIndex="min"/>
    </RollingFile>

Which doesn’t seem to break anything, but I don’t know where the rotated logs end up. I can see that my journal file gets rotated (I usually check the file size every day and sometimes it’s shrunk drastically). Any thoughts?

Edit:
My graylog messages are stored in /var/lib/graylog-server/journal/messagejournal-0/000*.log

4

He

the journal is not the logfile - never touch anything that is inside the journal folder. The journal is the internal buffer of Graylog that is used when Graylog is not able to ingest messages to elasticsearch.

The logfiles are by default in /var/log/graylog-server/server.log (might be different depending how you install, the link to the docs and default files above!)

5

Ah, so since it’s the buffer that would explain the… shrinkage.
In that case, does the log4j config look correct to you?

Edit:
I’m pretty confused now. I have the server.log file, but it seems to just be internal messages from the server and not the messages received - it also hasn’t been appended in over a week despite me receiving a few hundred thousand syslog messages since then. I know the Journal file changes in real time, but if it’s just a buffer does that mean messages are not being saved anywhere? For reference…

Message journal files /var/lib/graylog-server/journal
Log Files /var/log/graylog-server/

Are both correct, but the former is a buffer and the latter seems to be internal messages only…does graylog require additional configuration to keep messages?

6

Sorry Jan. I found what I was looking for. It was in/var/log/upstart/graylog-server and I can see that in this directory there are compressed log versions indicating that they are being rotated. Thank for your patience.

7

@KuboMD

Messages, after they are processes by Graylog are stored in Elasticsearch. Not as plain text files on disk - that might give you the confusion?

8

That would explain my confusion! Where can I read about elasticsearch x graylog?

9

In the documentation: http://docs.graylog.org/en/2.4/pages/architecture.html

10

Sorry Jan, one last question:

ubuntu@osshonisyslog1s:~$ locate server.log
/var/log/graylog-server/server.log
/var/log/upstart/graylog-server.log.1.gz
/var/log/upstart/graylog-server.log.2.gz
/var/log/upstart/graylog-server.log.3.gz
/var/log/upstart/graylog-server.log.4.gz
/var/log/upstart/graylog-server.log.5.gz
/var/log/upstart/graylog-server.log.6.gz
/var/log/upstart/graylog-server.log.7.gz
ubuntu@osshonisyslog1s:~$

The “server.log” seems like a plaintext record of ingested messages.

ubuntu@osshonisyslog1s:/var/log/upstart$ ls -anh | grep graylog
-rw-r----- 1 0 0 21M Aug 16 13:19 graylog-server.log
-rw-r----- 1 0 0 4.2M Aug 16 06:25 graylog-server.log.1.gz
-rw-r----- 1 0 0 4.7M Aug 15 06:25 graylog-server.log.2.gz
-rw-r----- 1 0 0 4.9M Aug 14 06:25 graylog-server.log.3.gz
-rw-r----- 1 0 0 3.8M Aug 13 06:25 graylog-server.log.4.gz
-rw-r----- 1 0 0 3.0M Aug 12 06:24 graylog-server.log.5.gz
-rw-r----- 1 0 0 2.4M Aug 11 06:25 graylog-server.log.6.gz
-rw-r----- 1 0 0 2.8M Aug 10 06:25 graylog-server.log.7.gz

Judging by the modify time, the “graylog-server.log” (uncompressed) file is being updated in real time. In all of the documentation I’ve look at and the questions I’ve asked this directory has never come up. Is it meant to be here? I quite like it, as I can uncompress the logs from another day to read them if I need to.

11

The “official” locations of default installation are written here: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

What OS did you install Graylog? It looks like it uses upstart to controll daemons.

But, the incoming messages are never stored in plain on disk with Graylog.

12

Thanks for the reply, Jan.
I checked again this morning and these same plaintext logs have been rotated again.
I installed on Trusty Tahr. The contents of the log file looks like:

06:50:27.485 [outputbuffer-processor-executor-2] INFO org.graylog2.outputs.LoggingOutput - Writing message: source: 192.168.196.13 | message: %EARL-SW2-4_STBY-1-EXCESSIVE_PARITY_ERROR: EARL 0: Parity error detected in VRAM { sequence_number: 23200 | level: 1 | gl2_remote_ip: 192.168.196.13 | gl2_remote_port: 49831 | gl2_source_node: 4ef50f94-6c7a-4faa-8114-e3805dedbdc0 | _id: 5a1ab5c0-a20b-11e8-9879-000c2974c901 | gl2_source_input: 5b6afe985ad3412dbf38f45c | facility: local7 | timestamp: 2018-08-17T06:46:37.000-04:00 }

Which seems like it contains a syslog message and the Graylog overhead…
Any insight?

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.