Graylog 4.1 on RedHat 8 craches daily

Graylog Central (peer support)
1

Hi,

cant find out why my Graylog 4 installation crashes every night.
It seems like graylog looses connection to elasticsearch but logs aren’t telling me anything or i don’t know what to search for. A restart of graylog service normally puts everything back online but sometimes i have to restart elasticsearch to.

My server has 16GB RAM ( VMware vs )
6GB for elastic in jvm.options and 6GB for graylog in /etc/sysconfig/graylog-server

The server is a fresh install of RedHat 8 and Graylog 4.0 then Graylog is upgraded to 4.0.1
graylog-server-4.0.1-1.noarch
elasticsearch-oss-7.10.1-1.x86_64
mongodb-org-4.2.11-1.el8.x86_64

Please advice how to find out whats causing this.

Thanks!

1 Like
2

@Mattias If you are using Minimum Graylog setup (All Graylog component on the same node), you should increase your RAM resources of the server or reduce JVM of elasticsearch Graylog to avoid server crash. You can also check your /var/log/message log file where you will get the cause of it.

I hope this will helps you!

1 Like
3

@makarands
Yes, everything is on the same server. I now have reduced memory (jvm) for elasticsearch to 4GB. Lets see what happens to night :slight_smile:
Thanks!

1 Like
4

@makarands It died again during the night so it dident help lowering mem.

Regards //Mattias

1 Like
5

@Mattias, I hope you have restarted ES and Graylog services after changing the JVM value of Elasticsearch and Graylog to take an effect of it. If you have already done this, you need to check your server.log file, where we might get its exact cause.

1 Like
6

Yes, everything is restarted. Today a got an update of graylog-forwarder. Everything is up to date. Testing with 8GB jvm for elastic and 5GB for Graylog today. Will see tomorrow if the server lives or not.
Java version is java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64

I cant see why in the server log files but i think its Graylog that cant connect to elastic. Systemctl restart graylog-server makes all online again.

On frontpage i get

n is undefined

Stack Trace:

Can post a new one tomorrow.

1 Like
7

@makarands here is the stack trace.

n is undefined

Stack Trace:

XJonkqpu/cf<@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2362458
m/<@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2495772
value@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2496969
Hi@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:293781
Ri@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:293576
Ls@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:329190
Mc@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:320633
mc@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:320556
sc@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:317586
Za/<@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:269227
434/exports.unstable_runWithPriority@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:780503
Ua@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:268936
Za@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:269174
Va@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:269107
nc@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:314371
yi@https://SERVERNAME/assets/vendor.594b2a39cb22b445205e.js:2:286185
s/</<@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:6001
s@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2452063
o/<[22]</n.exports/T.prototype._settlePromiseFromHandler@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2424990
o/<[22]</n.exports/T.prototype._settlePromise@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2425790
o/<[22]</n.exports/T.prototype._settlePromise0@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2426489
o/<[22]</n.exports/T.prototype._settlePromises@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2427841
l@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2379002
c@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2378942
o/<[2]</a.prototype._drainQueues@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2380056
a/this.drainQueues@https://SERVERNAME/assets/app.a41ce78e1beaf9cde903.js:2:2378875

Component Stack:

in ConnectStoresWrapper[Provider] stores=view,configs
in Tu
in Zu
in Ju
in bu
in ConnectStoresWrapper[bu] stores=view
in lf
in Unknown
in hf
in mf
in g
in Unknown
in Unknown
in n
in div
in af
in t
in t
in div
in Lt
in o
in ft
in h
in Rt
in t
in t
in c
in t
in Ir
in Unknown
in ConnectStoresWrapper[Unknown/Anonymous] stores=streams
in je
in T
in f
in v
in b
in Unknown
in n
in F
in ConnectStoresWrapper[F] stores=currentUser,server,sessionId
in je
in T
in Z
1 Like
8

@Mattias Please check and share your server.log where you will cause of it.

1 Like
9

@makarands
Here is graylog-server log “server.log”
2021-01-05T08:46:58.404+01:00 INFO [ServerBootstrap] Graylog server up and running.
2021-01-05T08:46:58.487+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb24ff9c4713e500e05fb40] is now STARTING
2021-01-05T08:46:58.491+01:00 INFO [InputStateListener] Input [Beats/5fb25db31b25d227a81a9b8c] is now STARTING
2021-01-05T08:46:58.492+01:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5fb25dd51b25d227a81a9c40] is now STARTING
2021-01-05T08:46:58.493+01:00 INFO [InputStateListener] Input [Syslog TCP/5fb25041c4713e500e05fbae] is now STARTING
2021-01-05T08:46:58.493+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9ce7] is now STARTING
2021-01-05T08:46:58.494+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9d01] is now STARTING
2021-01-05T08:46:58.495+01:00 INFO [InputStateListener] Input [Beats/5fb278a7ddeeb038f1e76f7d] is now STARTING
2021-01-05T08:46:58.495+01:00 INFO [InputStateListener] Input [Syslog TCP/5fc76ce7ab03c4279ef1d1a2] is now STARTING
2021-01-05T08:46:58.501+01:00 INFO [InputStateListener] Input [GELF UDP/5fb25daf1b25d227a81a9b37] is now STARTING
2021-01-05T08:46:58.670+01:00 INFO [InputStateListener] Input [Beats/5fb25db31b25d227a81a9b8c] is now RUNNING
2021-01-05T08:46:58.680+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=TCP_WinDNS_1555, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x579b4a95, L:/0:0:0:0:0:0:0:0%0:1555]) should be 1048576 but is 425984.
2021-01-05T08:46:58.681+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Syslog_TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x60017a13, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-05T08:46:58.682+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Vmware_Syslog_TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x0dbae9b9, L:/0:0:0:0:0:0:0:0%0:2514]) should be 1048576 but is 425984.
2021-01-05T08:46:58.682+01:00 INFO [InputStateListener] Input [Beats/5fb278a7ddeeb038f1e76f7d] is now RUNNING
2021-01-05T08:46:58.684+01:00 INFO [InputStateListener] Input [Syslog TCP/5fc76ce7ab03c4279ef1d1a2] is now RUNNING
2021-01-05T08:46:58.686+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=BEATS, type=org.graylog.plugins.beats.Beats2Input, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x50cceb26, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984.
2021-01-05T08:46:58.686+01:00 INFO [InputStateListener] Input [Syslog TCP/5fb25041c4713e500e05fbae] is now RUNNING
2021-01-05T08:46:58.694+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x55b13d26, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-05T08:46:58.695+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog_UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xcf26fe81, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-05T08:46:58.697+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x36bc3656, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.697+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x6de63b49, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-05T08:46:58.715+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x7aa76e86, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-05T08:46:58.744+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x30da754b, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.747+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xb116461c, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-05T08:46:58.748+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xaf62b608, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-05T08:46:58.754+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog_UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x38252d07, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-05T08:46:58.744+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xd5833f4c, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-05T08:46:58.757+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb24ff9c4713e500e05fb40] is now RUNNING
2021-01-05T08:46:58.798+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x662ac9f4, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.798+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xe28a423c, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-05T08:46:58.803+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x3e543db0, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-05T08:46:58.804+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xc706df56, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-05T08:46:58.806+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xcf92329a, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-05T08:46:58.808+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xbbd576eb, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-05T08:46:58.807+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xa4e0542d, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.809+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x6150967d, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-05T08:46:58.813+01:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5fb25dd51b25d227a81a9c40] is now RUNNING
2021-01-05T08:46:58.813+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xcd492521, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.815+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9d01] is now RUNNING
2021-01-05T08:46:58.823+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xd271ddd7, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.825+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x1c585c26, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.827+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x7f96ebe1, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.828+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9ce7] is now RUNNING
2021-01-05T08:46:58.832+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xa21cc579, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.834+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x3e6ce961, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.835+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xd884074c, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.838+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x2b3ef8eb, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.839+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x2ea29815, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.840+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xab2b1d94, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.854+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x51ff4d07, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.857+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x4ee51b17, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.858+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xa1a3f798, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.865+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xf47dcd6f, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.867+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x3a0d45ee, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.869+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x9d807707, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.870+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xcdd1c87d, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.875+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xf244f4ea, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.878+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x1795c011, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.880+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xefa409c8, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-05T08:46:58.882+01:00 INFO [InputStateListener] Input [GELF UDP/5fb25daf1b25d227a81a9b37] is now RUNNING
2021-01-05T08:49:29.237+01:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:11}] to localhost:27017
2021-01-05T08:49:29.257+01:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:12}] to localhost:27017
2021-01-05T08:49:29.269+01:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:13}] to localhost:27017
2021-01-06T08:46:32.740+01:00 ERROR [FileInfo] Couldn’t get file info for path: /usr/local/etc/graylog/GeoLite2-City.mmdb
java.nio.file.NoSuchFileException: /usr/local/etc/graylog/GeoLite2-City.mmdb
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) ~[?:1.8.0_275]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:1.8.0_275]
at java.nio.file.Files.readAttributes(Files.java:1737) ~[?:1.8.0_275]
at org.graylog2.plugin.utilities.FileInfo.forPath(FileInfo.java:76) ~[graylog.jar:?]
at org.graylog2.plugin.utilities.FileInfo.checkForChange(FileInfo.java:96) ~[graylog.jar:?]
at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doRefresh(MaxmindDataAdapter.java:123) ~[graylog.jar:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.refresh(LookupDataAdapter.java:109) ~[graylog.jar:?]
at org.graylog2.lookup.LookupDataAdapterRefreshService.lambda$schedule$0(LookupDataAdapterRefreshService.java:142) ~[graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-01-06T08:46:32.747+01:00 WARN [MaxmindDataAdapter] Unable to load changed database file, leaving old one intact. Error message: /usr/local/etc/graylog/GeoLite2-City.mmdb (No such file or directory)

elastic log “graylog.log”
[2021-01-05T08:46:12,953][INFO ][o.e.n.Node ] [log01.kb.se] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-923087555446056225, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-01-05T08:46:14,934][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [aggs-matrix-stats]
[2021-01-05T08:46:14,935][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [analysis-common]
[2021-01-05T08:46:14,935][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [geo]
[2021-01-05T08:46:14,935][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [ingest-common]
[2021-01-05T08:46:14,935][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [ingest-geoip]
[2021-01-05T08:46:14,935][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [ingest-user-agent]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [kibana]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [lang-expression]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [lang-mustache]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [lang-painless]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [mapper-extras]
[2021-01-05T08:46:14,936][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [parent-join]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [percolator]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [rank-eval]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [reindex]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [repository-url]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [systemd]
[2021-01-05T08:46:14,937][INFO ][o.e.p.PluginsService ] [log01.kb.se] loaded module [transport-netty4]
[2021-01-05T08:46:14,938][INFO ][o.e.p.PluginsService ] [log01.kb.se] no plugins loaded
[2021-01-05T08:46:14,979][INFO ][o.e.e.NodeEnvironment ] [log01.kb.se] using [1] data paths, mounts [[/ (/dev/mapper/rhel_log01-root)]], net usable_space [461.9gb], net total_space [492.2gb], types [xfs]
[2021-01-05T08:46:14,979][INFO ][o.e.e.NodeEnvironment ] [log01.kb.se] heap size [8gb], compressed ordinary object pointers [true]
[2021-01-05T08:46:15,194][INFO ][o.e.n.Node ] [log01.kb.se] node name [log01.kb.se], node ID [1wwiVxGkRD-l0E0gwYrNhw], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]
[2021-01-05T08:46:18,128][INFO ][o.e.t.NettyAllocator ] [log01.kb.se] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-01-05T08:46:18,188][INFO ][o.e.d.DiscoveryModule ] [log01.kb.se] using discovery type [zen] and seed hosts providers [settings]
[2021-01-05T08:46:18,419][WARN ][o.e.g.DanglingIndicesState] [log01.kb.se] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-01-05T08:46:18,565][INFO ][o.e.n.Node ] [log01.kb.se] initialized
[2021-01-05T08:46:18,565][INFO ][o.e.n.Node ] [log01.kb.se] starting …
[2021-01-05T08:46:18,669][INFO ][o.e.t.TransportService ] [log01.kb.se] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-01-05T08:46:18,889][INFO ][o.e.c.c.Coordinator ] [log01.kb.se] cluster UUID [D_XeMfbpT5KKq7h2vxZcUg]
[2021-01-05T08:46:19,011][INFO ][o.e.c.s.MasterService ] [log01.kb.se] elected-as-master ([1] nodes joined)[{log01.kb.se}{1wwiVxGkRD-l0E0gwYrNhw}{uyP69rb1Tnqb3FMuELARpw}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 36, version: 659, delta: master node changed {previous , current [{log01.kb.se}{1wwiVxGkRD-l0E0gwYrNhw}{uyP69rb1Tnqb3FMuELARpw}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-01-05T08:46:19,099][INFO ][o.e.c.s.ClusterApplierService] [log01.kb.se] master node changed {previous , current [{log01.kb.se}{1wwiVxGkRD-l0E0gwYrNhw}{uyP69rb1Tnqb3FMuELARpw}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 36, version: 659, reason: Publication{term=36, version=659}
[2021-01-05T08:46:19,124][INFO ][o.e.h.AbstractHttpServerTransport] [log01.kb.se] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-01-05T08:46:19,125][INFO ][o.e.n.Node ] [log01.kb.se] started
[2021-01-05T08:46:19,453][INFO ][o.e.g.GatewayService ] [log01.kb.se] recovered [7] indices into cluster_state
[2021-01-05T08:46:21,646][INFO ][o.e.c.r.a.AllocationService] [log01.kb.se] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]]]).

elastic graylog_server.json
{“type”: “server”, “timestamp”: “2021-01-05T08:46:12,944+01:00”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “version[7.10.1], pid[1689], build[oss/rpm/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/4.18.0-240.8.1.el8_3.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:12,952+01:00”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:12,953+01:00”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-923087555446056225, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,934+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [aggs-matrix-stats]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,935+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [analysis-common]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,935+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [geo]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,935+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [ingest-common]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,935+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [ingest-geoip]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,935+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [ingest-user-agent]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [kibana]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [lang-expression]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [lang-mustache]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [lang-painless]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [mapper-extras]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,936+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [parent-join]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,937+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [percolator]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,937+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [rank-eval]” }
{“type”: “server”, “timestamp”: “2021-01-05T08:46:14,937+01:00”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “log01.kb.se”, “message”: “loaded module [reindex]” }
…

1 Like
10

Some hints from logs:

  1. Missing file, check that you provided correct path in geo lookup table configuration:
    /usr/local/etc/graylog/GeoLite2-City.mmdb
  2. Logs are from morning, not night in which graylog crashed, please check also nightly logs
  3. There is weird OpenJDK 15 version in logs JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9], check installed java versions. Don’t you use it also for graylog, which only support java 8/11.
1 Like
11

@shoothub
Hi, thanks for answering!

  1. That seems to be hard coded somehow into graylog-bin or im changing values in the wrong file. I dont have /usr/local/etc/graylog…
    Under web-gui system/configuration GEO location was disabled. I have now enabled it and pointed it to the right destination but i still get that graylog cant find it in /usr/local/etc/graylog/GeoLite2-City.mmdb even after restart of graylog.
  2. Dont exactly know when graylog dies… sometime between 5pm an 8am :slight_smile: Cant find anything in the logs that tells me that graylog is not functioning. Can i upload the complete log files somewhere?
  3. I dont use openjdk 15 for what i can see.

rpm -qa | grep java
java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64
java-1.8.0-openjdk-devel-1.8.0.275.b01-1.el8_3.x86_64
tzdata-java-2020d-1.el8.noarch
javapackages-filesystem-5.3.0-1.module+el8+2447+6f56d9a6.noarch
java-1.8.0-openjdk-headless-1.8.0.275.b01-1.el8_3.x86_64

rpm -qa | grep openjdk
java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64
java-1.8.0-openjdk-devel-1.8.0.275.b01-1.el8_3.x86_64
java-1.8.0-openjdk-headless-1.8.0.275.b01-1.el8_3.x86_64

Thanks!!!
Regards Mattias

1 Like
12

well it died again sometime after working hours…

1 Like
13

What’s your actual java heap size for graylog and elastic? Try to lower graylog heap size to default value and check.

1 Like
14

in /etc/sysconfig/graylog-server i have:

Default Java options for heap and garbage collection.

GRAYLOG_SERVER_JAVA_OPTS="-Xms6g -Xmx6g

in /etc/elasticsearch/jvm.options i have:
-Xms6g
-Xmx6g

VS Machine has 16GB memory.
I have tested with 2/4/6 for both elastic and graylog. Still crashes daily.

image
image1207×114 10.6 KB

1 Like
15

OK,

  1. Do you have new log from last crash?
  2. After crash, what is status of graylog service?
    systemctl status graylog-server.service
  3. After crash, what is status of elastic service?
    systemctl status elastic.service
  4. Is graylog still listening on port 9000 after crash?
    netstat -tupln
1 Like
16

@shoothub
Here is “server.log” from when i started graylog yesterday and till i restarted the service this morning.
I hade to make the dir /usr/local/etc/graylog and create a symlink for the mmdb file pointing to correct location. What i fill in into gui doesnt matter.

I will get back to you on Monday with the rest of the info.

b25daf1b25d227a81a9b2f [@22db4e65] STARTING
2021-01-07T15:44:08.532+01:00 INFO [LookupTableService] Data Adapter user-agent-parser/5fb25dce1b25d227a81a9bff [@417393b7] RUNNING
2021-01-07T15:44:08.533+01:00 ERROR [FileInfo] Couldn’t get file info for path: /etc/graylog/server/GeoLite2-City.mmdb
java.nio.file.NoSuchFileException: /etc/graylog/server/GeoLite2-City.mmdb
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) ~[?:1.8.0_275]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:1.8.0_275]
at java.nio.file.Files.readAttributes(Files.java:1737) ~[?:1.8.0_275]
at org.graylog2.plugin.utilities.FileInfo.forPath(FileInfo.java:76) [graylog.jar:?]
at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doStart(MaxmindDataAdapter.java:88) [graylog.jar:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:79) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-01-07T15:44:08.534+01:00 WARN [MaxmindDataAdapter] Cannot read database file /etc/graylog/server/GeoLite2-City.mmdb
2021-01-07T15:44:08.536+01:00 INFO [LookupTableService] Data Adapter geolite2-city/5fb25deb1b25d227a81a9d38 [@3923459a] RUNNING
2021-01-07T15:44:08.535+01:00 ERROR [FileInfo] Couldn’t get file info for path: /usr/local/etc/graylog/GeoLite2-City.mmdb
java.nio.file.NoSuchFileException: /usr/local/etc/graylog/GeoLite2-City.mmdb
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_275]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:1.8.0_275]
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) ~[?:1.8.0_275]
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:1.8.0_275]
at java.nio.file.Files.readAttributes(Files.java:1737) ~[?:1.8.0_275]
at org.graylog2.plugin.utilities.FileInfo.forPath(FileInfo.java:76) [graylog.jar:?]
at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doStart(MaxmindDataAdapter.java:88) [graylog.jar:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:79) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-01-07T15:44:08.537+01:00 WARN [MaxmindDataAdapter] Cannot read database file /usr/local/etc/graylog/GeoLite2-City.mmdb
2021-01-07T15:44:08.539+01:00 INFO [LookupTableService] Data Adapter geo-ip-city-maxmind-adapater/5fb25daf1b25d227a81a9b2f [@22db4e65] RUNNING
2021-01-07T15:44:08.539+01:00 INFO [LookupDataAdapterRefreshService] Adding job for geo-ip-city-maxmind-adapater/5fb25daf1b25d227a81a9b2f/@22db4e65 [interval=86400000ms]
2021-01-07T15:44:08.632+01:00 INFO [LookupTableService] Cache geolite2-city/5fb25deb1b25d227a81a9d39 [@cd42595] STARTING
2021-01-07T15:44:08.633+01:00 INFO [LookupTableService] Cache geo-ip-city-maxmind-cache/5fb25daf1b25d227a81a9b25 [@346a3133] STARTING
2021-01-07T15:44:08.639+01:00 INFO [LookupTableService] Cache geolite2-city/5fb25deb1b25d227a81a9d39 [@cd42595] RUNNING
2021-01-07T15:44:08.643+01:00 INFO [LookupTableService] Cache geo-ip-city-maxmind-cache/5fb25daf1b25d227a81a9b25 [@346a3133] RUNNING
2021-01-07T15:44:08.640+01:00 INFO [LookupTableService] Cache useragent-cacher/5fb25dce1b25d227a81a9c00 [@73619ff] STARTING
2021-01-07T15:44:08.644+01:00 INFO [LookupTableService] Cache useragent-cacher/5fb25dce1b25d227a81a9c00 [@73619ff] RUNNING
2021-01-07T15:44:08.670+01:00 INFO [LookupTableService] Starting lookup table user_agent_lookup/5fb25dce1b25d227a81a9c01 [@2c84f4f4] using cache useragent-cacher/5fb25dce1b25d227a81a9c00 [@73619ff], data adapter user-agent-parser/5fb25dce1b25d227a81a9bff [@417393b7]
2021-01-07T15:44:33.917+01:00 INFO [NetworkListener] Started listener bound to [10.50.16.49:9000]
2021-01-07T15:44:33.918+01:00 INFO [HttpServer] [HttpServer] Started.
2021-01-07T15:44:33.918+01:00 INFO [JerseyService] Started REST API at <10.50.16.49:9000>
2021-01-07T15:44:33.919+01:00 INFO [ServiceManagerListener] Services are healthy
2021-01-07T15:44:33.920+01:00 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, UrlWhitelistService [RUNNING]=149, JournalReader [RUNNING]=149, OutputSetupService [RUNNING]=149, BufferSynchronizerService [RUNNING]=150, GracefulShutdownService [RUNNING]=151, KafkaJournal [RUNNING]=167, JobSchedulerService [RUNNING]=180, MongoDBProcessingStatusRecorderService [RUNNING]=197, ConfigurationEtagService [RUNNING]=241, EtagService [RUNNING]=245, PeriodicalsService [RUNNING]=387, StreamCacheService [RUNNING]=433, LookupTableService [RUNNING]=540, JerseyService [RUNNING]=25791}
2021-01-07T15:44:33.920+01:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-01-07T15:44:33.928+01:00 INFO [ServerBootstrap] Graylog server up and running.
2021-01-07T15:44:34.147+01:00 INFO [InputStateListener] Input [GELF UDP/5fb25daf1b25d227a81a9b37] is now STARTING
2021-01-07T15:44:34.151+01:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5fb25dd51b25d227a81a9c40] is now STARTING
2021-01-07T15:44:34.168+01:00 INFO [InputStateListener] Input [Beats/5fb25db31b25d227a81a9b8c] is now STARTING
2021-01-07T15:44:34.173+01:00 INFO [InputStateListener] Input [Syslog TCP/5fb25041c4713e500e05fbae] is now STARTING
2021-01-07T15:44:34.174+01:00 INFO [InputStateListener] Input [Beats/5fb278a7ddeeb038f1e76f7d] is now STARTING
2021-01-07T15:44:34.184+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb24ff9c4713e500e05fb40] is now STARTING
2021-01-07T15:44:34.186+01:00 INFO [InputStateListener] Input [Syslog TCP/5fc76ce7ab03c4279ef1d1a2] is now STARTING
2021-01-07T15:44:34.189+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9d01] is now STARTING
2021-01-07T15:44:34.190+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9ce7] is now STARTING
2021-01-07T15:44:34.342+01:00 INFO [InputStateListener] Input [Syslog TCP/5fc76ce7ab03c4279ef1d1a2] is now RUNNING
2021-01-07T15:44:34.351+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Vmware_Syslog_TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x692750c1, L:/0:0:0:0:0:0:0:0%0:2514]) should be 1048576 but is 425984.
2021-01-07T15:44:34.353+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=TCP_WinDNS_1555, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x13981a1c, L:/0:0:0:0:0:0:0:0%0:1555]) should be 1048576 but is 425984.
2021-01-07T15:44:34.353+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Syslog_TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x0d08ecea, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-07T15:44:34.353+01:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=BEATS, type=org.graylog.plugins.beats.Beats2Input, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xd9172de2, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984.
2021-01-07T15:44:34.353+01:00 INFO [InputStateListener] Input [Syslog TCP/5fb25041c4713e500e05fbae] is now RUNNING
2021-01-07T15:44:34.357+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog_UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xcb5385d8, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-07T15:44:34.358+01:00 INFO [InputStateListener] Input [Beats/5fb25db31b25d227a81a9b8c] is now RUNNING
2021-01-07T15:44:34.362+01:00 INFO [InputStateListener] Input [Beats/5fb278a7ddeeb038f1e76f7d] is now RUNNING
2021-01-07T15:44:34.369+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xab6d55ab, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.369+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xcabe22ae, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-07T15:44:34.370+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x1df2aaa4, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-07T15:44:34.514+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xc705435e, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-07T15:44:34.523+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog_UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x835307e0, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-01-07T15:44:34.527+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb24ff9c4713e500e05fb40] is now RUNNING
2021-01-07T15:44:34.545+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x61cd6455, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-07T15:44:34.546+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x953ef7e4, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-07T15:44:34.546+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0x946d6c0d, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-07T15:44:34.547+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x5519f01b, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.574+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0xb89a1100, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-07T15:44:34.585+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xfe3afc76, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-07T15:44:34.589+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x63c6cdee, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-07T15:44:34.595+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx access log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x8f4ee75d, L:/0:0:0:0:0:0:0:0%0:12304]) should be 1048576 but is 425984.
2021-01-07T15:44:34.595+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=54672bc0-337c-4ffc-92ba-fa8419a91009} (channel [id: 0xb5ff80db, L:/0:0:0:0:0:0:0:0%0:15514]) should be 262144 but is 425984.
2021-01-07T15:44:34.598+01:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/5fb25dd51b25d227a81a9c40] is now RUNNING
2021-01-07T15:44:34.599+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9d01] is now RUNNING
2021-01-07T15:44:34.601+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xab34c2bd, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.612+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=nginx error log, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x7b6cfe53, L:/0:0:0:0:0:0:0:0%0:12305]) should be 1048576 but is 425984.
2021-01-07T15:44:34.619+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x02be9fb4, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.621+01:00 INFO [InputStateListener] Input [Syslog UDP/5fb25de71b25d227a81a9ce7] is now RUNNING
2021-01-07T15:44:34.660+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xf1784698, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.690+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xb0338d6e, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.707+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x702907ee, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.722+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xe714c9ac, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.743+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x5d9fe1e4, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.752+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x95efd924, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.760+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x678009d7, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.778+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xbf03ac00, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.781+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x149e6db7, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.786+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xc007d9bd, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.794+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x61f6b8fe, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.814+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xa017485b, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.867+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x5a200150, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.881+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x1a491dc6, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.923+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x81e6512a, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.940+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xb4564f5b, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.980+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x35fdaf5f, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:35.007+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x2308f3d3, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:34.989+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0xd7ff7cef, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:35.016+01:00 WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=nginx logs, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} (channel [id: 0x92c75344, L:/0:0:0:0:0:0:0:0%0:12401]) should be 1048576 but is 425984.
2021-01-07T15:44:35.025+01:00 INFO [InputStateListener] Input [GELF UDP/5fb25daf1b25d227a81a9b37] is now RUNNING
2021-01-07T15:44:35.459+01:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:65}] to localhost:27017
2021-01-07T15:44:35.464+01:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:68}] to localhost:27017
2021-01-07T15:44:35.464+01:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:67}] to localhost:27017
2021-01-07T15:44:35.475+01:00 INFO [connection] Opened connection [connectionId{localValue:12, serverValue:70}] to localhost:27017
2021-01-07T15:44:35.476+01:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:69}] to localhost:27017
2021-01-07T15:44:35.482+01:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:66}] to localhost:27017
2021-01-08T08:10:32.602+01:00 INFO [Server] SIGNAL received. Shutting down.
2021-01-08T08:10:32.629+01:00 INFO [GracefulShutdown] Graceful shutdown initiated.

1 Like
17

@shoothub
Here is the rest…

● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-01-08 08:22:46 CET; 3 days ago
Docs: http://docs.graylog.org/
Main PID: 43441 (graylog-server)
Tasks: 235 (limit: 100498)
Memory: 3.2G
CGroup: /system.slice/graylog-server.service
├─43441 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─43460 /usr/bin/java -Xms6g -Xmx6g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm /usr/share/graylog-server/graylog.jar server -f /etc/graylog/server/server.conf -np

Jan 08 08:22:46 log01.kb.se systemd[1]: Started Graylog server.
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running) since Fri 2021-01-08 14:16:43 CET; 2 days ago
Docs: https://www.elastic.co
Main PID: 46895 (java)
Tasks: 64 (limit: 100498)
Memory: 9.0G
CGroup: /system.slice/elasticsearch.service
└─46895 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms6g -Xmx6g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-12619449312765848683 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=3221225472 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=oss -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

Jan 08 14:16:34 log01.kb.se systemd[1]: Starting Elasticsearch…
Jan 08 14:16:43 log01.kb.se systemd[1]: Started Elasticsearch.
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.50.16.49:19999 0.0.0.0:* LISTEN 1711/netdata
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 1093/nrpe
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 1692/snmpd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1397/mongod
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 1098/memcached
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1091/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1651/master
tcp 0 0 127.0.0.1:8125 0.0.0.0:* LISTEN 1711/netdata
tcp6 0 0 :::5666 :::* LISTEN 1093/nrpe
tcp6 0 0 10.50.16.49:9000 :::* LISTEN 43460/java
tcp6 0 0 :::1514 :::* LISTEN 43460/java
tcp6 0 0 ::1:11211 :::* LISTEN 1098/memcached
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 46895/java
tcp6 0 0 ::1:9200 :::* LISTEN 46895/java
tcp6 0 0 :::80 :::* LISTEN 1099/httpd
tcp6 0 0 :::2514 :::* LISTEN 43460/java
tcp6 0 0 :::1555 :::* LISTEN 43460/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 46895/java
tcp6 0 0 ::1:9300 :::* LISTEN 46895/java
tcp6 0 0 :::5044 :::* LISTEN 43460/java
tcp6 0 0 :::22 :::* LISTEN 1091/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1651/master
tcp6 0 0 :::443 :::* LISTEN 1099/httpd
tcp6 0 0 ::1:8125 :::* LISTEN 1711/netdata
udp 0 0 0.0.0.0:58304 0.0.0.0:* 1692/snmpd
udp 0 0 127.0.0.1:8125 0.0.0.0:* 1711/netdata
udp 0 0 0.0.0.0:111 0.0.0.0:* 1/systemd
udp 0 0 0.0.0.0:161 0.0.0.0:* 1692/snmpd
udp 0 0 127.0.0.1:323 0.0.0.0:* 1038/chronyd
udp6 0 0 :::1514 :::* 43460/java
udp6 0 0 :::1514 :::* 43460/java
udp6 0 0 :::12304 :::* 43460/java
udp6 0 0 :::12304 :::* 43460/java
udp6 0 0 :::12304 :::* 43460/java
udp6 0 0 :::12304 :::* 43460/java
udp6 0 0 :::12305 :::* 43460/java
udp6 0 0 :::12305 :::* 43460/java
udp6 0 0 :::12305 :::* 43460/java
udp6 0 0 :::12305 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::12401 :::* 43460/java
udp6 0 0 :::15514 :::* 43460/java
udp6 0 0 :::15514 :::* 43460/java
udp6 0 0 :::15514 :::* 43460/java
udp6 0 0 :::15514 :::* 43460/java
udp6 0 0 ::1:8125 :::* 1711/netdata
udp6 0 0 :::111 :::* 1/systemd
udp6 0 0 ::1:323 :::* 1038/chronyd

18

“Cant connect to localhost 9000/tcp”
Ah found that my server was set to listen on ip:9000 so my test was wrong. Will check it again at next crach.

19

@Mattias as sort of a housekeeping item, please ensure you surround code output with ```. This will make it easier on helping to read.

1 Like
20

@shoothub
Thanks for answering. Do you have any more ideas? Should i change java version to Oracle java? Should i start anything i debug mode? Im all out of ideas after messing with this for two months.
I need to make a decision if we are gonna go ahead with Graylog or move back to Kibana.
I like the gui in Graylog but itś to unstable.
Thanks!!