Graylog filling up syslog and kern.log at alarming rate

Rob · March 28, 2017, 5:49pm

I have set up Graylog in AWS using the ami provided by them. Everything is set up and messages are coming in but I am having diskspace issues due to Graylog filling up /var/log/kern.log and /var/logs/syslog at a ridiculous rate. I am talking about a couple gb every few minutes.

Has anyone seen this problem before and know how to address it? All the messages appear to be the same thing over and over. Just looks like internal logging. See below for an example:

Kern.log:

Mar 28 16:53:00 graylog kernel [    0.000000] KERNEL supported cpus:
Mar 28 16:53:00 graylog kernel [    0.000000]   Intel GenuineIntel
Mar 28 16:53:00 graylog kernel [    0.000000]   AMD AuthenticAMD
Mar 28 16:53:00 graylog kernel [    0.000000]   Centaur CentaurHauls
Mar 28 16:53:00 graylog kernel [    0.000000] e820: BIOS-provided physical RAM map:
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000040fffffff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] NX (Execute Disable) protection: active
Mar 28 16:53:00 graylog kernel [    0.000000] SMBIOS 2.4 present.
Mar 28 16:53:00 graylog kernel [    0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 11/11/2016
Mar 28 16:53:00 graylog kernel [    0.000000] Hypervisor detected: Xen HVM
Mar 28 16:53:00 graylog kernel [    0.000000] Xen version 4.2.
Mar 28 16:53:00 graylog kernel [    0.000000] Xen Platform PCI: I/O protocol version 1
Mar 28 16:53:00 graylog kernel [    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
Mar 28 16:53:00 graylog kernel [    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
Mar 28 16:53:00 graylog kernel [    0.000000] You might have to change the root device
Mar 28 16:53:00 graylog kernel [    0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
Mar 28 16:53:00 graylog kernel [    0.000000] in your root= kernel command line option
Mar 28 16:53:00 graylog kernel [    0.000000] HVMOP_pagetable_dying not supported
Mar 28 16:53:00 graylog kernel [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Mar 28 16:53:00 graylog kernel [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] No AGP bridge found
Mar 28 16:53:00 graylog kernel [    0.000000] e820: last_pfn = 0x410000 max_arch_pfn = 0x400000000
Mar 28 16:53:00 graylog kernel [    0.000000] MTRR fixed ranges enabled:
Mar 28 16:53:00 graylog kernel [    0.000000]   00000-9FFFF write-back
Mar 28 16:53:00 graylog kernel [    0.000000]   A0000-BFFFF write-combining
Mar 28 16:53:00 graylog kernel [    0.000000]   C0000-FFFFF write-back
Mar 28 16:53:00 graylog kernel [    0.000000] MTRR variable ranges enabled:
Mar 28 16:53:00 graylog kernel [    0.000000]   0 base 0000F0000000 mask 3FFFF8000000 uncachable
Mar 28 16:53:00 graylog kernel [    0.000000]   1 base 0000F8000000 mask 3FFFFC000000 uncachable
Mar 28 16:53:00 graylog kernel [    0.000000]   2 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   3 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   4 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   5 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   6 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   7 disabled
Mar 28 16:53:00 graylog kernel [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Mar 28 16:53:00 graylog kernel [    0.000000] e820: last_pfn = 0xf0000 max_arch_pfn = 0x400000000
Mar 28 16:53:00 graylog kernel [    0.000000] found SMP MP-table at [mem 0x000fbc80-0x000fbc8f] mapped at [ffff8800000fbc80]
Mar 28 16:53:00 graylog kernel [    0.000000] Scanning 1 areas for low memory corruption
Mar 28 16:53:00 graylog kernel [    0.000000] Base memory trampoline at [ffff880000098000] 98000 size 24576
Mar 28 16:53:00 graylog kernel [    0.000000] Using GB pages for direct mapping
Mar 28 16:53:00 graylog kernel [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]

Syslog:

Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 17:00:50 graylog kernel [ Mar 28 17:00:50 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog Mar 28 16:53:00 graylog kernel [ Mar 28 17:17:01 graylog Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog Mar 28 16:53:29 graylog Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ Mar 28 16:53:00 graylog kernel [ 0.000000] nr_irqs_gsi: 64
0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
0.000000] e820: [mem 0xf0000000-0xfbffffff] available for PCI devices
0.000000] Booting paravirtualized kernel on Xen HVM
0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:15 nr_node_ids:2
0.000000] PERCPU: Embedded 27 pages/cpu @ffff880207c00000 s81536 r8192 d20864 u262144
0.000000] pcpu-alloc: s81536 r8192 d20864 u262144 alloc=1*2097152
520.173393] Setting capacity to 419430400
520.173401] xvda: detected capacity change from 107374182400 to 214748364800
0.000000] Memory: 16422576K/16776820K available (7432K kernel code, 1147K rwdata, 3424K rodata, 1340K init, 1448K bss, 354244K reserved)
0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=2
0.000000] Hierarchical RCU implementation.
0.000000] #011RCU dyntick-idle grace-period acceleration is enabled.
0.000000] #011RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=15.
0.000000] #011Offload RCU callbacks from all CPUs
0.000000] #011Offload RCU callbacks from CPUs: 0-14.
0.000000] NR_IRQS:16640 nr_irqs:1208 16
0.000000] xen:events: Xen HVM callback vector for event delivery is enabled
0.000000] Console: colour VGA+ 80x25
0.000000] console [tty1] enabled
0.000000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
0.000000] console [ttyS0] enabled
0.000000] allocated 67108864 bytes of page_cgroup
0.000000] please try ‘cgroup_disable=memory’ option if you don’t want memory cgroups
0.000000] tsc: Detected 2300.086 MHz processor
0.008000] Calibrating delay loop (skipped), value calculated using timer frequency… 4600.17 BogoMIPS (lpj=9200344)
acpid starting up with netlink and the input layer
0.009778] pid_max: default: 32768 minimum: 301
CRON[2232] (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
0.012049] Security Framework initialized
0.014408] AppArmor: AppArmor initialized
0.016002] Yama: becoming mindful.
0.020913] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
0.028731] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
0.040289] Initializing cgroup subsys memory
0.044009] Initializing cgroup subsys devices
0.048002] Initializing cgroup subsys freezer
0.050556] Initializing cgroup subsys blkio
0.052002] Initializing cgroup subsys perf_event
pollinate system was previously seeded at [2017-01-30 13:42:21.287396999 +0000]
ntpd[1702] ntpd 4.2.6p5@1.2349-o Wed Oct 5 12:35:25 UTC 2016 (1)
0.056004] Initializing cgroup subsys hugetlb
0.058577] CPU: Physical Processor ID: 0
0.060003] CPU: Processor Core ID: 0

jtkarvo · March 29, 2017, 3:36pm

These logs are just normal logs. The server booted 16.53. Normally boot logs do not grow to gigabytes; you should post logs from a later time to have a comment on what actually is logged there.

danclark · May 28, 2017, 11:21pm

Hmm, mines doing the same, screaming through the same log entries over and over.
filled 90GB of log space overnight!

danclark · May 28, 2017, 11:39pm

ah, it’s this entry we put in rsyslog.conf

#. @127.0.0.1:514;RSYSLOG_SyslogProtocol23Format

It’s causing logs to duplicate and fly up their own arse…

Topic		Replies	Views
Journal is filling up along the time Graylog Central (peer support)	2	860	March 30, 2017
Journal utilization is too high again Graylog Central (peer support)	7	6068	June 15, 2018
Graylog server rebooting frequently Graylog Central (peer support)	4	1386	June 24, 2021
Graylog Sizing/Optimzation Graylog Central (peer support) sidecar , nxlog , nodatanx	4	6420	December 15, 2017
Again Graylog is backed up and slow to write out Graylog Central (peer support)	2	875	April 20, 2019

Graylog filling up syslog and kern.log at alarming rate

Related topics