Graylog filling up syslog and kern.log at alarming rate


#1

I have set up Graylog in AWS using the ami provided by them. Everything is set up and messages are coming in but I am having diskspace issues due to Graylog filling up /var/log/kern.log and /var/logs/syslog at a ridiculous rate. I am talking about a couple gb every few minutes.

Has anyone seen this problem before and know how to address it? All the messages appear to be the same thing over and over. Just looks like internal logging. See below for an example:

Kern.log:

Mar 28 16:53:00 graylog kernel [    0.000000] KERNEL supported cpus:
Mar 28 16:53:00 graylog kernel [    0.000000]   Intel GenuineIntel
Mar 28 16:53:00 graylog kernel [    0.000000]   AMD AuthenticAMD
Mar 28 16:53:00 graylog kernel [    0.000000]   Centaur CentaurHauls
Mar 28 16:53:00 graylog kernel [    0.000000] e820: BIOS-provided physical RAM map:
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
Mar 28 16:53:00 graylog kernel [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000040fffffff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] NX (Execute Disable) protection: active
Mar 28 16:53:00 graylog kernel [    0.000000] SMBIOS 2.4 present.
Mar 28 16:53:00 graylog kernel [    0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 11/11/2016
Mar 28 16:53:00 graylog kernel [    0.000000] Hypervisor detected: Xen HVM
Mar 28 16:53:00 graylog kernel [    0.000000] Xen version 4.2.
Mar 28 16:53:00 graylog kernel [    0.000000] Xen Platform PCI: I/O protocol version 1
Mar 28 16:53:00 graylog kernel [    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
Mar 28 16:53:00 graylog kernel [    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
Mar 28 16:53:00 graylog kernel [    0.000000] You might have to change the root device
Mar 28 16:53:00 graylog kernel [    0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
Mar 28 16:53:00 graylog kernel [    0.000000] in your root= kernel command line option
Mar 28 16:53:00 graylog kernel [    0.000000] HVMOP_pagetable_dying not supported
Mar 28 16:53:00 graylog kernel [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Mar 28 16:53:00 graylog kernel [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
Mar 28 16:53:00 graylog kernel [    0.000000] No AGP bridge found
Mar 28 16:53:00 graylog kernel [    0.000000] e820: last_pfn = 0x410000 max_arch_pfn = 0x400000000
Mar 28 16:53:00 graylog kernel [    0.000000] MTRR fixed ranges enabled:
Mar 28 16:53:00 graylog kernel [    0.000000]   00000-9FFFF write-back
Mar 28 16:53:00 graylog kernel [    0.000000]   A0000-BFFFF write-combining
Mar 28 16:53:00 graylog kernel [    0.000000]   C0000-FFFFF write-back
Mar 28 16:53:00 graylog kernel [    0.000000] MTRR variable ranges enabled:
Mar 28 16:53:00 graylog kernel [    0.000000]   0 base 0000F0000000 mask 3FFFF8000000 uncachable
Mar 28 16:53:00 graylog kernel [    0.000000]   1 base 0000F8000000 mask 3FFFFC000000 uncachable
Mar 28 16:53:00 graylog kernel [    0.000000]   2 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   3 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   4 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   5 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   6 disabled
Mar 28 16:53:00 graylog kernel [    0.000000]   7 disabled
Mar 28 16:53:00 graylog kernel [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Mar 28 16:53:00 graylog kernel [    0.000000] e820: last_pfn = 0xf0000 max_arch_pfn = 0x400000000
Mar 28 16:53:00 graylog kernel [    0.000000] found SMP MP-table at [mem 0x000fbc80-0x000fbc8f] mapped at [ffff8800000fbc80]
Mar 28 16:53:00 graylog kernel [    0.000000] Scanning 1 areas for low memory corruption
Mar 28 16:53:00 graylog kernel [    0.000000] Base memory trampoline at [ffff880000098000] 98000 size 24576
Mar 28 16:53:00 graylog kernel [    0.000000] Using GB pages for direct mapping
Mar 28 16:53:00 graylog kernel [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]

Syslog:

Mar 28 16:53:00 graylog kernel [ 0.000000] nr_irqs_gsi: 64
Mar 28 16:53:00 graylog kernel [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
Mar 28 16:53:00 graylog kernel [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
Mar 28 16:53:00 graylog kernel [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
Mar 28 16:53:00 graylog kernel [ 0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
Mar 28 16:53:00 graylog kernel [ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
Mar 28 16:53:00 graylog kernel [ 0.000000] e820: [mem 0xf0000000-0xfbffffff] available for PCI devices
Mar 28 16:53:00 graylog kernel [ 0.000000] Booting paravirtualized kernel on Xen HVM
Mar 28 16:53:00 graylog kernel [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:15 nr_node_ids:2
Mar 28 16:53:00 graylog kernel [ 0.000000] PERCPU: Embedded 27 pages/cpu @ffff880207c00000 s81536 r8192 d20864 u262144
Mar 28 16:53:00 graylog kernel [ 0.000000] pcpu-alloc: s81536 r8192 d20864 u262144 alloc=1*2097152
Mar 28 17:00:50 graylog kernel [ 520.173393] Setting capacity to 419430400
Mar 28 17:00:50 graylog kernel [ 520.173401] xvda: detected capacity change from 107374182400 to 214748364800
Mar 28 16:53:00 graylog kernel [ 0.000000] Memory: 16422576K/16776820K available (7432K kernel code, 1147K rwdata, 3424K rodata, 1340K init, 1448K bss, 354244K reserved)
Mar 28 16:53:00 graylog kernel [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=2
Mar 28 16:53:00 graylog kernel [ 0.000000] Hierarchical RCU implementation.
Mar 28 16:53:00 graylog kernel [ 0.000000] #011RCU dyntick-idle grace-period acceleration is enabled.
Mar 28 16:53:00 graylog kernel [ 0.000000] #011RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=15.
Mar 28 16:53:00 graylog kernel [ 0.000000] #011Offload RCU callbacks from all CPUs
Mar 28 16:53:00 graylog kernel [ 0.000000] #011Offload RCU callbacks from CPUs: 0-14.
Mar 28 16:53:00 graylog kernel [ 0.000000] NR_IRQS:16640 nr_irqs:1208 16
Mar 28 16:53:00 graylog kernel [ 0.000000] xen:events: Xen HVM callback vector for event delivery is enabled
Mar 28 16:53:00 graylog kernel [ 0.000000] Console: colour VGA+ 80x25
Mar 28 16:53:00 graylog kernel [ 0.000000] console [tty1] enabled
Mar 28 16:53:00 graylog kernel [ 0.000000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
Mar 28 16:53:00 graylog kernel [ 0.000000] console [ttyS0] enabled
Mar 28 16:53:00 graylog kernel [ 0.000000] allocated 67108864 bytes of page_cgroup
Mar 28 16:53:00 graylog kernel [ 0.000000] please try ‘cgroup_disable=memory’ option if you don’t want memory cgroups
Mar 28 16:53:00 graylog kernel [ 0.000000] tsc: Detected 2300.086 MHz processor
Mar 28 16:53:00 graylog kernel [ 0.008000] Calibrating delay loop (skipped), value calculated using timer frequency… 4600.17 BogoMIPS (lpj=9200344)
Mar 28 16:53:00 graylog acpid starting up with netlink and the input layer
Mar 28 16:53:00 graylog kernel [ 0.009778] pid_max: default: 32768 minimum: 301
Mar 28 17:17:01 graylog CRON[2232] (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Mar 28 16:53:00 graylog kernel [ 0.012049] Security Framework initialized
Mar 28 16:53:00 graylog kernel [ 0.014408] AppArmor: AppArmor initialized
Mar 28 16:53:00 graylog kernel [ 0.016002] Yama: becoming mindful.
Mar 28 16:53:00 graylog kernel [ 0.020913] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mar 28 16:53:00 graylog kernel [ 0.028731] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Mar 28 16:53:00 graylog kernel [ 0.040289] Initializing cgroup subsys memory
Mar 28 16:53:00 graylog kernel [ 0.044009] Initializing cgroup subsys devices
Mar 28 16:53:00 graylog kernel [ 0.048002] Initializing cgroup subsys freezer
Mar 28 16:53:00 graylog kernel [ 0.050556] Initializing cgroup subsys blkio
Mar 28 16:53:00 graylog kernel [ 0.052002] Initializing cgroup subsys perf_event
Mar 28 16:53:00 graylog pollinate system was previously seeded at [2017-01-30 13:42:21.287396999 +0000]
Mar 28 16:53:29 graylog ntpd[1702] ntpd 4.2.6p5@1.2349-o Wed Oct 5 12:35:25 UTC 2016 (1)
Mar 28 16:53:00 graylog kernel [ 0.056004] Initializing cgroup subsys hugetlb
Mar 28 16:53:00 graylog kernel [ 0.058577] CPU: Physical Processor ID: 0
Mar 28 16:53:00 graylog kernel [ 0.060003] CPU: Processor Core ID: 0


#2

These logs are just normal logs. The server booted 16.53. Normally boot logs do not grow to gigabytes; you should post logs from a later time to have a comment on what actually is logged there.


(Dan Clark) #3

Hmm, mines doing the same, screaming through the same log entries over and over.
filled 90GB of log space overnight!


(Dan Clark) #4

ah, it’s this entry we put in rsyslog.conf

#. @127.0.0.1:514;RSYSLOG_SyslogProtocol23Format

It’s causing logs to duplicate and fly up their own arse…