Graylog 4 craches every day

Graylog Central (peer support)
1 
Hello,
every night my Graylog 4 installation craches. When trying to login the next day i get "can not load streams". Have looked at logfiles so many times but cant find out why.

Server is RHEL8 in Vmware with 16GB memory.
elasticsearch-oss 7 has 6GB assigned
graylog has 4GB assigned

Here is trace:

n is undefined

Stack Trace:

XJonkqpu/cf<@https://SERVER/assets/app.a41ce78e1beaf9cde903.js:2:2362458
m/<@https://SERVER/assets/app.a41ce78e1beaf9cde903.js:2:2495772
value@https://SERVER/assets/app.a41ce78e1beaf9cde903.js:2:2496969
Hi@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:293781
Ri@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:293576
Ls@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:329190
Mc@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:320633
mc@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:320556
sc@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:317586
Za/<@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:269227
434/exports.unstable_runWithPriority@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:780503
Ua@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:268936
Za@https://SERVERassets/vendor.594b2a39cb22b445205e.js:2:269174
Va@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:269107
nc@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:314371
yi@https://SERVER/assets/vendor.594b2a39cb22b445205e.js:2:286185
t/JgW3b2/r.a/</<@https://SERVER/assets/NewSearchPage.a41ce78e1beaf9cde903.js:1:3055


Component Stack:

    in ConnectStoresWrapper[Provider] stores=view,configs
    in Tu
    in Zu
    in Ju
    in bu
    in ConnectStoresWrapper[bu] stores=view
    in lf
    in Unknown
    in hf
    in mf
    in b
    in Unknown
    in n
    in Unknown
    in n
    in t
    in t
    in div
    in Lt
    in o
    in ft
    in h
    in Rt
    in t
    in t
    in c
    in t
    in Ir
    in Unknown
    in ConnectStoresWrapper[Unknown/Anonymous] stores=streams
    in je
    in T
    in f
    in v
    in b
    in Unknown
    in n
    in F
    in ConnectStoresWrapper[F] stores=currentUser,server,sessionId
    in je
    in T
    in Z

What can it be?
Can i include more log files?
Suggestions?

Thanks all!!!
2

Hello @Mattias,welcome!

How are you resolving the issue each time it happens? It says it cannot load streams – how have you been correcting that?

3

Hi ttsandrew,
a restart of graylog server fixes it but it happens again within 24 hours.

4

Found this!

ERROR [ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it’s garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:385)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
java.lang.Thread.run(Thread.java:748)

5

Hey @Mattias, do you mean a restart of the graylog-server service or you’re restarting the host?

What’s the status of your mongod service? Any errors in the logs?

6

Morning,

a "systemctl restart graylog-server " puts the server online again.

The only problem i could find in mongod.log was this, now fixed.

2020-12-17T23:50:55.236+0100 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2020-12-18T07:49:41.157+0100 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.

I wonder if this could be my problem. https://github.com/Graylog2/graylog2-server/issues/9637

Regards Mattias

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.