Upgrading from 4.3.3

Graylog Central (peer support)
1

1. Describe your incident:
Hello, I am running 4.3.3 fine in Docker with ES and mongoDB,
as per docker-compose in the examples.
However when I try to update from 4.3.3 to anything newer I get the errors

2. Describe your environment:

  • OS Information:
    Host running Debian 10.2.1-6, Docker version 20.10.9

Intel 6th gen i7, 50gigs RAM

  • Service logs, configurations, and environment variables:
graylog          | #
graylog          | # There is insufficient memory for the Java Runtime Environment to continue.
graylog          | # Cannot create GC thread. Out of system resources.
graylog          | # Can not save log file, dump to screen..
graylog          | #
graylog          | # There is insufficient memory for the Java Runtime Environment to continue.
graylog          | # Cannot create GC thread. Out of system resources.
graylog          | # Possible reasons:
graylog          | #   The system is out of physical RAM or swap space
graylog          | #   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
graylog          | # Possible solutions:
graylog          | #   Reduce memory load on the system
graylog          | #   Increase physical memory or swap space
graylog          | #   Check if swap backing store is full
graylog          | #   Decrease Java heap size (-Xmx/-Xms)
graylog          | #   Decrease number of Java threads
graylog          | #   Decrease Java thread stack sizes (-Xss)
graylog          | #   Set larger code cache with -XX:ReservedCodeCacheSize=
graylog          | # This output file may be truncated or incomplete.
graylog          | #
graylog          | #  Out of Memory Error (gcTaskThread.cpp:48), pid=7, tid=0x00007f33cd93db80
graylog          | #
graylog          | # JRE version:  (8.0_342-b07) (build )
graylog          | # Java VM: OpenJDK 64-Bit Server VM (25.342-b07 mixed mode linux-amd64 compressed oops)
graylog          | # Core dump written. Default location: /usr/share/graylog/core or core.7
graylog          | #
graylog          |
graylog          | ---------------  T H R E A D  ---------------
graylog          |
graylog          | Current thread (0x00005628c33ca800):  JavaThread "Unknown thread" [_thread_in_vm, id=7, stack(0x00007ffc500eb000,0x00007ffc501eb000)]
graylog          |
graylog          | Stack: [0x00007ffc500eb000,0x00007ffc501eb000],  sp=0x00007ffc501e4230,  free space=996k
graylog          | Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
graylog          | V  [libjvm.so+0xb39a32]  VMError::report_and_die()+0x2d2
graylog          | V  [libjvm.so+0x4e4917]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
graylog          | V  [libjvm.so+0x5cb24f]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0x13f
graylog          | V  [libjvm.so+0x5ca3bd]  GCTaskManager::initialize()+0x36d
graylog          | V  [libjvm.so+0x992b02]  ParallelScavengeHeap::initialize()+0x3a2
graylog          | V  [libjvm.so+0xafcffa]  Universe::initialize_heap()+0x16a
graylog          | V  [libjvm.so+0xafd2e3]  universe_init()+0x33
graylog          | V  [libjvm.so+0x642e3d]  init_globals()+0x5d
graylog          | V  [libjvm.so+0xadfbe1]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x4f1
graylog          | V  [libjvm.so+0x720f30]  JNI_CreateJavaVM+0x50
graylog          | C  [libjli.so+0x7eb0]  JavaMain+0x90
graylog          | C  [libjli.so+0xd8e9]  ContinueInNewThread0+0x69
graylog          | C  [libjli.so+0x9c72]  ContinueInNewThread+0x52
graylog          | C  [libjli.so+0xa380]  JLI_Launch+0x5c0
graylog          | C  [java+0x651]  main+0x41
graylog          | C  [libc.so.6+0x29d90]
graylog          |
graylog          |
graylog          | ---------------  P R O C E S S  ---------------
graylog          |
graylog          | Java Threads: ( => current thread )
graylog          |
graylog          | Other Threads:
graylog          |
graylog          | =>0x00005628c33ca800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=7, stack(0x00007ffc500eb000,0x00007ffc501eb000)]
graylog          |
graylog          | VM state:not at safepoint (not fully initialized)
graylog          |
graylog          | VM Mutex/Monitor currently owned by a thread: None
graylog          |
graylog          | heap address: 0x0000000501200000, size: 12022 MB, Compressed Oops mode: Non-zero based:0x00000005011ff000, Oop shift amount: 3
graylog          | Narrow klass base: 0x0000000000000000, Narrow klass shift: 0
graylog          |
graylog          | GC Heap History (0 events):
graylog          | No events
graylog          |
graylog          | Deoptimization events (0 events):
graylog          | No events
graylog          |
graylog          | Classes redefined (0 events):
graylog          | No events
graylog          |
graylog          | Internal exceptions (0 events):
graylog          | No events
graylog          |
graylog          | Events (0 events):
graylog          | No events
graylog          |
graylog          |
graylog          | Dynamic libraries:
graylog          | 501200000-518a00000 rw-p 00000000 00:00 0
graylog          | 518a00000-678d00000 ---p 00000000 00:00 0
graylog          | 678d00000-690500000 rw-p 00000000 00:00 0
graylog          | 690500000-7f0800000 ---p 00000000 00:00 0
graylog          | 5628c1c00000-5628c1c01000 r-xp 00000000 08:01 9051749                    /opt/java/openjdk/bin/java
graylog          | 5628c1e00000-5628c1e01000 r--p 00000000 08:01 9051749                    /opt/java/openjdk/bin/java
graylog          | 5628c1e01000-5628c1e02000 rw-p 00001000 08:01 9051749                    /opt/java/openjdk/bin/java
graylog          | 5628c33ad000-5628c3411000 rw-p 00000000 00:00 0                          [heap]
graylog          | 7f33b9f06000-7f33b9fc2000 rw-p 00000000 00:00 0
graylog          | 7f33b9fc2000-7f33baac4000 ---p 00000000 00:00 0
graylog          | 7f33baac4000-7f33bab80000 rw-p 00000000 00:00 0
graylog          | 7f33bab80000-7f33bb681000 ---p 00000000 00:00 0
graylog          | 7f33bb681000-7f33bb73e000 rw-p 00000000 00:00 0
graylog          | 7f33bb73e000-7f33bc23f000 ---p 00000000 00:00 0
graylog          | 7f33bc23f000-7f33bc24a000 rw-p 00000000 00:00 0
graylog          | 7f33bc24a000-7f33bc600000 ---p 00000000 00:00 0
graylog          | 7f33bc600000-7f33bc870000 rwxp 00000000 00:00 0
graylog          | 7f33bc870000-7f33cb600000 ---p 00000000 00:00 0
graylog          | 7f33cb600000-7f33cb61b000 r-xp 00000000 08:01 19795252                   /opt/java/openjdk/lib/amd64/libzip.so
graylog          | 7f33cb61b000-7f33cb81a000 ---p 0001b000 08:01 19795252                   /opt/java/openjdk/lib/amd64/libzip.so
graylog          | 7f33cb81a000-7f33cb81b000 r--p 0001a000 08:01 19795252                   /opt/java/openjdk/lib/amd64/libzip.so
graylog          | 7f33cb81b000-7f33cb81c000 rw-p 0001b000 08:01 19795252                   /opt/java/openjdk/lib/amd64/libzip.so
graylog          | 7f33cba00000-7f33cba29000 r-xp 00000000 08:01 19795232                   /opt/java/openjdk/lib/amd64/libjava.so
graylog          | 7f33cba29000-7f33cbc28000 ---p 00029000 08:01 19795232                   /opt/java/openjdk/lib/amd64/libjava.so
graylog          | 7f33cbc28000-7f33cbc29000 r--p 00028000 08:01 19795232                   /opt/java/openjdk/lib/amd64/libjava.so
graylog          | 7f33cbc29000-7f33cbc2b000 rw-p 00029000 08:01 19795232                   /opt/java/openjdk/lib/amd64/libjava.so
graylog          | 7f33cbc5e000-7f33cbe00000 rw-p 00000000 00:00 0
graylog          | 7f33cbe00000-7f33cbe0d000 r-xp 00000000 08:01 19795251                   /opt/java/openjdk/lib/amd64/libverify.so
graylog          | 7f33cbe0d000-7f33cc00c000 ---p 0000d000 08:01 19795251                   /opt/java/openjdk/lib/amd64/libverify.so
graylog          | 7f33cc00c000-7f33cc00e000 r--p 0000c000 08:01 19795251                   /opt/java/openjdk/lib/amd64/libverify.so
graylog          | 7f33cc00e000-7f33cc00f000 rw-p 0000e000 08:01 19795251                   /opt/java/openjdk/lib/amd64/libverify.so
graylog          | 7f33cc10c000-7f33cc200000 rw-p 00000000 00:00 0
graylog          | 7f33cc200000-7f33ccf5e000 r-xp 00000000 08:01 19795256                   /opt/java/openjdk/lib/amd64/server/libjvm.so
graylog          | 7f33ccf5e000-7f33cd15d000 ---p 00d5e000 08:01 19795256                   /opt/java/openjdk/lib/amd64/server/libjvm.so
graylog          | 7f33cd15d000-7f33cd1f4000 r--p 00d5d000 08:01 19795256                   /opt/java/openjdk/lib/amd64/server/libjvm.so
graylog          | 7f33cd1f4000-7f33cd21d000 rw-p 00df4000 08:01 19795256                   /opt/java/openjdk/lib/amd64/server/libjvm.so
graylog          | 7f33cd21d000-7f33cd250000 rw-p 00000000 00:00 0
graylog          | 7f33cd2d7000-7f33cd2d8000 ---p 00000000 00:00 0
graylog          | 7f33cd2d8000-7f33cd3d8000 rw-p 00000000 00:00 0
graylog          | 7f33cd3d8000-7f33cd400000 r--p 00000000 08:01 19667194                   /usr/lib/x86_64-linux-gnu/libc.so.6
graylog          | 7f33cd400000-7f33cd595000 r-xp 00028000 08:01 19667194                   /usr/lib/x86_64-linux-gnu/libc.so.6
graylog          | 7f33cd595000-7f33cd5ed000 r--p 001bd000 08:01 19667194                   /usr/lib/x86_64-linux-gnu/libc.so.6
graylog          | 7f33cd5ed000-7f33cd5f1000 r--p 00214000 08:01 19667194                   /usr/lib/x86_64-linux-gnu/libc.so.6
graylog          | 7f33cd5f1000-7f33cd5f3000 rw-p 00218000 08:01 19667194                   /usr/lib/x86_64-linux-gnu/libc.so.6
graylog          | 7f33cd5f3000-7f33cd600000 rw-p 00000000 00:00 0
graylog          | 7f33cd600000-7f33cd617000 r-xp 00000000 08:01 19795219                   /opt/java/openjdk/lib/amd64/jli/libjli.so
graylog          | 7f33cd617000-7f33cd816000 ---p 00017000 08:01 19795219                   /opt/java/openjdk/lib/amd64/jli/libjli.so
graylog          | 7f33cd816000-7f33cd817000 r--p 00016000 08:01 19795219                   /opt/java/openjdk/lib/amd64/jli/libjli.so
graylog          | 7f33cd817000-7f33cd818000 rw-p 00017000 08:01 19795219                   /opt/java/openjdk/lib/amd64/jli/libjli.so
graylog          | 7f33cd849000-7f33cd851000 rw-p 00000000 00:00 0
graylog          | 7f33cd851000-7f33cd852000 r--p 00000000 08:01 19667291                   /usr/lib/x86_64-linux-gnu/librt.so.1
graylog          | 7f33cd852000-7f33cd853000 r-xp 00001000 08:01 19667291                   /usr/lib/x86_64-linux-gnu/librt.so.1
graylog          | 7f33cd853000-7f33cd854000 r--p 00002000 08:01 19667291                   /usr/lib/x86_64-linux-gnu/librt.so.1
graylog          | 7f33cd854000-7f33cd855000 r--p 00002000 08:01 19667291                   /usr/lib/x86_64-linux-gnu/librt.so.1
graylog          | 7f33cd855000-7f33cd856000 rw-p 00003000 08:01 19667291                   /usr/lib/x86_64-linux-gnu/librt.so.1
graylog          | 7f33cd856000-7f33cd864000 r--p 00000000 08:01 19667246                   /usr/lib/x86_64-linux-gnu/libm.so.6
graylog          | 7f33cd864000-7f33cd8e0000 r-xp 0000e000 08:01 19667246                   /usr/lib/x86_64-linux-gnu/libm.so.6
graylog          | 7f33cd8e0000-7f33cd93b000 r--p 0008a000 08:01 19667246                   /usr/lib/x86_64-linux-gnu/libm.so.6
graylog          | 7f33cd93b000-7f33cd93c000 r--p 000e4000 08:01 19667246                   /usr/lib/x86_64-linux-gnu/libm.so.6
graylog          | 7f33cd93c000-7f33cd93d000 rw-p 000e5000 08:01 19667246                   /usr/lib/x86_64-linux-gnu/libm.so.6
graylog          | 7f33cd93d000-7f33cd93f000 rw-p 00000000 00:00 0
graylog          | 7f33cd93f000-7f33cd940000 r--p 00000000 08:01 19667208                   /usr/lib/x86_64-linux-gnu/libdl.so.2
graylog          | 7f33cd940000-7f33cd941000 r-xp 00001000 08:01 19667208                   /usr/lib/x86_64-linux-gnu/libdl.so.2
graylog          | 7f33cd941000-7f33cd942000 r--p 00002000 08:01 19667208                   /usr/lib/x86_64-linux-gnu/libdl.so.2
graylog          | 7f33cd942000-7f33cd943000 r--p 00002000 08:01 19667208                   /usr/lib/x86_64-linux-gnu/libdl.so.2
graylog          | 7f33cd943000-7f33cd944000 rw-p 00003000 08:01 19667208                   /usr/lib/x86_64-linux-gnu/libdl.so.2
graylog          | 7f33cd944000-7f33cd945000 r--p 00000000 08:01 19667289                   /usr/lib/x86_64-linux-gnu/libpthread.so.0
graylog          | 7f33cd945000-7f33cd946000 r-xp 00001000 08:01 19667289                   /usr/lib/x86_64-linux-gnu/libpthread.so.0
graylog          | 7f33cd946000-7f33cd947000 r--p 00002000 08:01 19667289                   /usr/lib/x86_64-linux-gnu/libpthread.so.0
graylog          | 7f33cd947000-7f33cd948000 r--p 00002000 08:01 19667289                   /usr/lib/x86_64-linux-gnu/libpthread.so.0
graylog          | 7f33cd948000-7f33cd949000 rw-p 00003000 08:01 19667289                   /usr/lib/x86_64-linux-gnu/libpthread.so.0
graylog          | 7f33cd94b000-7f33cd94e000 rw-p 00000000 00:00 0
graylog          | 7f33cd94e000-7f33cd950000 r--p 00000000 08:01 19667176                   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
graylog          | 7f33cd950000-7f33cd97a000 r-xp 00002000 08:01 19667176                   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
graylog          | 7f33cd97a000-7f33cd985000 r--p 0002c000 08:01 19667176                   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
graylog          | 7f33cd985000-7f33cd986000 r--p 00000000 00:00 0
graylog          | 7f33cd986000-7f33cd988000 r--p 00037000 08:01 19667176                   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
graylog          | 7f33cd988000-7f33cd98a000 rw-p 00039000 08:01 19667176                   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
graylog          | 7ffc500eb000-7ffc500ee000 ---p 00000000 00:00 0
graylog          | 7ffc500ee000-7ffc501eb000 rw-p 00000000 00:00 0                          [stack]
graylog          | 7ffc501ef000-7ffc501f3000 r--p 00000000 00:00 0                          [vvar]
graylog          | 7ffc501f3000-7ffc501f5000 r-xp 00000000 00:00 0                          [vdso]
graylog          |
graylog          | VM Arguments:
graylog          | jvm_args: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
graylog          | java_command: /usr/share/graylog/graylog.jar server -f /usr/share/graylog/data/config/graylog.conf
graylog          | java_class_path (initial): /usr/share/graylog/graylog.jar
graylog          | Launcher Type: SUN_STANDARD
graylog          |
graylog          | Environment Variables:
graylog          | JAVA_HOME=/opt/java/openjdk
graylog          | PATH=/usr/share/graylog/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
graylog          |
graylog          | Signal Handlers:
graylog          | SIGSEGV: [libjvm.so+0xb3b0d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGBUS: [libjvm.so+0xb3b0d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGFPE: [libjvm.so+0x963540], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGPIPE: [libjvm.so+0x963540], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGXFSZ: [libjvm.so+0x963540], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGILL: [libjvm.so+0x963540], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
graylog          | SIGUSR2: [libjvm.so+0x963af0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
graylog          | SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
graylog          | SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
graylog          | SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
graylog          | SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
graylog          |
graylog          |
graylog          | ---------------  S Y S T E M  ---------------
graylog          |
graylog          | OS:DISTRIB_ID=Ubuntu
graylog          | DISTRIB_RELEASE=22.04
graylog          | DISTRIB_CODENAME=jammy
graylog          | DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
graylog          |
graylog          | uname:Linux 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64
graylog          | libc:glibc 2.35 NPTL 2.35
graylog          | rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
graylog          | load average:0.72 2.07 1.80
graylog          |
graylog          | /proc/meminfo:
graylog          | MemTotal:       49235732 kB
graylog          | MemFree:        30577328 kB
graylog          | MemAvailable:   41621804 kB
graylog          | Buffers:         1609152 kB
graylog          | Cached:          8783056 kB
graylog          | SwapCached:            0 kB
graylog          | Active:          3644104 kB
graylog          | Inactive:       11604136 kB
graylog          | Active(anon):      27708 kB
graylog          | Inactive(anon):  5072876 kB
graylog          | Active(file):    3616396 kB
graylog          | Inactive(file):  6531260 kB
graylog          | Unevictable:     1549056 kB
graylog          | Mlocked:         1548940 kB
graylog          | SwapTotal:             0 kB
graylog          | SwapFree:              0 kB
graylog          | Dirty:            138168 kB
graylog          | Writeback:             0 kB
graylog          | AnonPages:       6328744 kB
graylog          | Mapped:          1062900 kB
graylog          | Shmem:             70796 kB
graylog          | KReclaimable:    1496836 kB
graylog          | Slab:            1668332 kB
graylog          | SReclaimable:    1496836 kB
graylog          | SUnreclaim:       171496 kB
graylog          | KernelStack:       27408 kB
graylog          | PageTables:        44068 kB
graylog          | NFS_Unstable:          0 kB
graylog          | Bounce:                0 kB
graylog          | WritebackTmp:          0 kB
graylog          | CommitLimit:    24617864 kB
graylog          | Committed_AS:   23322868 kB
graylog          | VmallocTotal:   34359738367 kB
graylog          | VmallocUsed:       54196 kB
graylog          | VmallocChunk:          0 kB
graylog          | Percpu:             7872 kB
graylog          | HardwareCorrupted:     0 kB
graylog          | AnonHugePages:   4806656 kB
graylog          | ShmemHugePages:        0 kB
graylog          | ShmemPmdMapped:        0 kB
graylog          | FileHugePages:         0 kB
graylog          | FilePmdMapped:         0 kB
graylog          | HugePages_Total:       0
graylog          | HugePages_Free:        0
graylog          | HugePages_Rsvd:        0
graylog          | HugePages_Surp:        0
graylog          | Hugepagesize:       2048 kB
graylog          | Hugetlb:               0 kB
graylog          | DirectMap4k:      552592 kB
graylog          | DirectMap2M:    25548800 kB
graylog          | DirectMap1G:    24117248 kB
graylog          |
graylog          |
graylog          | CPU:total 8 (initial active 8) (4 cores per cpu, 2 threads per core) family 6 model 94 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
graylog          |
graylog          | /proc/cpuinfo:
graylog          | processor	: 0
graylog          | vendor_id	: GenuineIntel
graylog          | cpu family	: 6
graylog          | model		: 94
graylog          | model name	: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
graylog          | stepping	: 3
graylog          | microcode	: 0xea
graylog          | cpu MHz		: 3696.773
graylog          | cache size	: 8192 KB
graylog          | physical id	: 0
graylog          | siblings	: 8
graylog          | core id		: 0
graylog          | cpu cores	: 4
graylog          | apicid		: 0
graylog          | initial apicid	: 0
graylog          | fpu		: yes
graylog          | fpu_exception	: yes
graylog          | cpuid level	: 22
graylog          | wp		: yes
graylog          | flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
graylog          | vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs pml
graylog          | bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds
graylog          | bogomips	: 6799.81
graylog          | clflush size	: 64
graylog          | cache_alignment	: 64
graylog          | address sizes	: 39 bits physical, 48 bits virtual
graylog          | power management:
graylog          |
graylog          | processor	: 1
graylog          | vendor_id	: GenuineIntel
graylog          | cpu family	: 6
graylog          | model		: 94
graylog          | model name	: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
graylog          | stepping	: 3
graylog          | microcode	: 0xea
graylog          | cpu MHz		: 3655.685
graylog          | cache size	: 8192 KB
graylog          | physical id	: 0
graylog          | siblings	: 8
graylog          | core id		: 1
graylog          | cpu cores	: 4
graylog          | apicid		: 2
graylog          | initial apicid	: 2
graylog          | fpu		: yes
graylog          | fpu_exception	: yes
graylog          | cpuid level	: 22
graylog          | wp		: yes
----------     Intel CPU thingies -------
graylog          | bogomips	: 6799.81
graylog          | clflush size	: 64
graylog          | cache_alignment	: 64
graylog          | address sizes	: 39 bits physical, 48 bits virtual
graylog          | power management:
graylog          |
graylog          |
graylog          |
graylog          | Memory: 4k page, physical 49235732k(30577328k free), swap 0k(0k free)
graylog          |
graylog          | vm_info: OpenJDK 64-Bit Server VM (25.342-b07) for linux-amd64 JRE (1.8.0_342-b07), built on Jul 22 2022 08:47:00 by "jenkins" with gcc 7.5.0
graylog          |
graylog          | time: Tue Sep 27 08:56:18 2022
graylog          | timezone: UTC
graylog          | elapsed time: 0.005725 seconds (0d 0h 0m 0s)
graylog          |

3. What steps have you already taken to try and solve the problem?

Anything > 4.3.3 is not going to start up for me :confused:
I have no idea what is going wrong here,
maybe someone can see more than me.

4. How can the community help?

Thank you :stuck_out_tongue:

2

Hello @gotschi

first 26 lines of that log show “out of memory”.
Could you explain what is actually going on? Is graylog starting or does it fail to start?
How did you perform the upgrade process?
What does your Docker-compose file look like?

Any more details would help

-Thanks

3

Thank you for responding,
attached is my docker-compose file.

This is happening when i start up the containers with graylog > 4.3.3
everything runs and starts smooth with 4.3.3 but when I change the container image of graylog to 4.3.4 or higher I get these Errors and Graylog does not come up - it fails to start.

I have 50 gigs of ram, 90% of which is free on the Host and available for graylog if it wants so the error really makes no sense to me and I have no idea how to upgrade from 4.3.3 now

everything runs fine with 4.3.3, just not higher

version: "2.2"

services:
  mongodb:
    container_name: mongodb
    image: "mongo:4.2"
    volumes:
      - "/media/satahdd/docker-graylog/mongodb_data:/data/db"
    restart: "always"

  elasticsearch:
    container_name: elasticsearch
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
      bootstrap.memory_lock: "true"
      discovery.type: "single-node"
      http.host: "0.0.0.0"
      action.auto_create_index: "false"
    image: "docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2"
    ulimits:
      memlock:
        hard: -1
        soft: -1
    volumes:
      - "/media/satahdd/docker-graylog/es_data:/usr/share/elasticsearch/data"
    restart: "always"

  graylog:
    image: "graylog/graylog:4.3.3"
    container_name: graylog
    depends_on:
      elasticsearch:
        condition: "service_started"
      mongodb:
        condition: "service_started"
    entrypoint: "/usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh"
    environment:
      GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/config/node-id"
      GRAYLOG_PASSWORD_SECRET: ...
      GRAYLOG_ROOT_PASSWORD_SHA2: ...
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:7000"
      GRAYLOG_HTTP_EXTERNAL_URI: "http://localhost:7000/"
      GRAYLOG_ELASTICSEARCH_HOSTS: "http://elasticsearch:9200"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
    ports:
    - "5044:5044/tcp"   # Beats
    - "5140:5140/udp"   # Syslog
    - "5140:5140/tcp"   # Syslog
    - "5141:5141/udp"   # Opnsense Syslog
    - "5142:5142/udp"   # OpenWRT Syslog
    - "5555:5555/tcp"   # RAW TCP
    - "5555:5555/udp"   # RAW TCP
    - "7000:7000/tcp"   # Server API
    - "12201:12201/tcp" # GELF TCP
    - "12201:12201/udp" # GELF UDP
    - "12401:12401/udp"
    #- "10000:10000/tcp" # Custom TCP port
    #- "10000:10000/udp" # Custom UDP port
    - "13301:13301/tcp" # Forwarder data
    - "13302:13302/tcp" # Forwarder config
    volumes:
      - ./GeoLite2-City.mmdb:/etc/graylog/server/GeoLite2-City.mmdb
      - "/media/satahdd/docker-graylog/graylog_data:/usr/share/graylog/data/data"
      - "/media/satahdd/docker-graylog/graylog_journal:/usr/share/graylog/data/journal"
    restart: "always"
4

Nice thank you for the config,

I see you re-configure Graylogs default port from 9000 to 7000 :thinking:

What does the Graylog Docker logs show?

docker logs -f <container_id>

I just upgrade my Graylog Docker container, I just pulled new image and replace the name in my compose file. I think the logs may show what going on.

elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2-amd64
    # image: opensearchproject/opensearch:1.3.2
    network_mode: bridge

    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0    
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g   
  graylog:
    #image: graylog/graylog-enterprise:4.3.3-jre11 <--- Old
    image: graylog/graylog-enterprise:4.3.5-jre11 <--- new
    network_mode: bridge
    dns:
      - 10.10.10.12
      - 10.10.10.13
   # journal and config directories in local NFS share for persistence
    volumes:       
       - graylog_bin:/usr/share/graylog/bin
       - graylog_data:/usr/share/graylog/data/config
       - graylog_log:/usr/share/graylog/data/log
       - graylog_plugin:/usr/share/graylog/data/plugin
       - graylog_content:/usr/share/graylog/data/contentpacks
      

       #- ./graylog/data/journal:/usr/share/graylog/data/journal
       #- ./graylog/config:/usr/share/graylog/data/config
    environment:
      # Container time Zone
      - TZ=America/Chicago
      # CHANGE ME (must be at least 16 characters)!
5

Here is the output with 4.3.5-jre11: (fails)
https://x0.at/kB3v.txt

Here is the output with 4.3.3-jre11: (successful)
https://x0.at/NEGO.txt

Guess I’ll just stay on 4.3.3 forever and hide the notification

6

Have you tried to increase those settings? if you have the resources perhaps something like this
ES_JAVA_OPTS: “-Xms2g -Xmx2g”

Also have you seen this in your logs?

Reduce memory load on the system
graylog          | #   Increase physical memory or swap space
graylog          | #   Check if swap backing store is full
graylog          | #   Decrease Java heap size (-Xmx/-Xms)
graylog          | #   Decrease number of Java threads
graylog          | #   Decrease Java thread stack sizes (-Xss)
graylog          | #   Set larger code cache with -XX:ReservedCodeCacheSize=
graylog          | #   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
graylog          | #     placed in the first 4GB address space. The Java Heap base address is the
graylog          | #     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
graylog          | #     to set the Java Heap base and to place the Java Heap above 4GB virtual address.

Apart from that, you can override the JVM settings (i. e. the heap memory settings -Xms and -Xmx) with the GRAYLOG_SERVER_JAVA_OPTS environment variable.
I gues the moral of this issue is JAVA need more memory, that what I take from the logs.
You maybe able to set the file to something like this

# Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS="-Xms3g -Xmx3g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow "

Default files are located here

EDIT: Just another thought instead of stopping Mongo and ES leave those containers running and just stop Graylog container, swap out the image name and start the contain up.
Your logs looks different then mine do.

Did you use the following command?

root # docker logs -f <container_id>

Just a thought…

7

Thank you for your help but your suggestions don’t change a thing for this error.

It’s not elasticsearch I have problems with, it is graylog. changing Xms does not do a thing to Graylog in this case.
The other containers start up fine, everything works up until 4.3.3 and only then graylog fails, mongo and elastic are fine, no need to change anything there.

This is a Graylog bug and not a docker issue.

I used docker-compose up which gives you the output of all your starting containers FYI

docker logs -f does not change a thing about that either.

8

hello

Perhaps I didn’t explain this well.

You have two heap settings, One for Elasticsearch and One for Graylog (i.e JAVA) So in this case Your logs are complain about memory.

docker-compose up

Which show the info above, normally when troubleshooting issue/s we “tail log files” in this case using docker logs -f maybe help find something more we missed, just a thougt.

The thing is I just upgrade mine when you had this issue, I didn’t not have a problem with Graylog starting up ,nor did I have any information with lack of memory. The suggestions I offer is for troubleshooting and perhaps find something that was missed.

Understood, I was suggesting to NOT stop those container and leave them running since they are running fine, I was suggesting ONLY stopping the Graylog container replace the image and starting it back up, just for troubleshoot.

What I did notice in the compose file were changes in the configuration from default to some custom settings, not sure if that would make a impact. I also realize this was running fine before, maybe the new image does like custom settings, just a thought

EDIT: Did some digging around the web.
In your logs it shows a few of these errors before it fell on its face.

[warning][os,thread] Failed to start thread “GC Thread#0” - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached

Came across couple old post but had the same issue.

There were a couple solution in those not sure if they pertain your issue.

9

I did not expect this but it is indeed the issue from the first link,
I run on buster though so the only suspect left was docker itself…

ran apt upgrade docker - and tadaaaa 4.3.8 starts up without a hitch.

thank you very much, sorry for the rough response my patience went out

2 Likes
10

Awesome sauce man :+1: glad you fixed it.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.