Hello everyone, I am new here and also new to Graylog and Elasticsearch.
In short, because the title doesn’t say it all, the elasticsearch service is working sometimes and if i restart it (for some new configuration) or reboot the machine it stops and never stops again.
Here is the specs of my system and configuration:
System
- HP Proliant DL165 running esxi 6u3
- Graylog is running on VM Ubuntu 20.04.5 LTS (5.4.0) fully updated up to now
- VM has 16GB of RAM and 400GB of space
- VM has 2 interfaces on 2 different subnets to monitor
Configurations
In general i am trying this to work and all i did is the basics and defaults with a little bit of searching for proper functionality of the whole system.
One single VM contains Graylog AND Elasticsearch as well as Mongodb. The assets I am logging are 8 and they do not produce much of a traffic.
Graylog (top down of server.conf) version 4.3.7
- root_password_sha2 is set
- http_bind_address = 172.16.64.203:9000 (ipv6 is disabled)
- elasticsearch_hosts = http://172.16.64.203:9200
- mongodb_uri = mongodb://localhost/graylog (default)
- Xms1g and Xmx1g
Elasticsearch (top down of elasticsearch.yml) version 7.10.7
- node.name: Graylog
- network.host: 172.16.64.203
- http.port: 9200
- discovery.seed_hosts: [“Graylog”]
I also read that Elastic needs no more no less than half the RAM so the -Xms8g -Xmx8g settings are set to 8g in /etc/elasticsearch/jvm.options.d/jvm.options
Other information
openjdk version “11.0.16” 2022-07-19
Anything not mentioned is default.
After the first install according to Graylog docs everything is working fine. I have installed winlogbeat on a windows server and i am able to receive input. I also receive logs from another Syslog input from a linux server.
This is my second install so this time i left the VM running the whole night to make sure everything is working as expected. I restarted the Elasticsearch service but it never started again. Hitting a curl of http://172.16.64.203:9200 i get a connection refused. I also get a ERR_CONNECTION_REFUSED from the browser when i try to connect to Graylog web. And no, it is not a firewall issue since everything was working fine before the restart.
Any suggestions please? Any help is much appreciated. Thank you in advance.
I am attaching the log of the /var/log/elasticsearch/
[2022-09-22T11:55:50,783][INFO ][o.e.n.Node ] [Graylog] version[7.10.2], pid[5094], build[oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.4.0-126-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-09-22T11:55:50,789][INFO ][o.e.n.Node ] [Graylog] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-09-22T11:55:50,790][INFO ][o.e.n.Node ] [Graylog] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6532279290430572856, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6532279290430572856, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-09-22T11:55:52,523][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [aggs-matrix-stats]
[2022-09-22T11:55:52,524][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [analysis-common]
[2022-09-22T11:55:52,524][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [geo]
[2022-09-22T11:55:52,525][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [ingest-common]
[2022-09-22T11:55:52,525][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [ingest-geoip]
[2022-09-22T11:55:52,526][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [ingest-user-agent]
[2022-09-22T11:55:52,526][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [kibana]
[2022-09-22T11:55:52,527][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [lang-expression]
[2022-09-22T11:55:52,527][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [lang-mustache]
[2022-09-22T11:55:52,528][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [lang-painless]
[2022-09-22T11:55:52,528][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [mapper-extras]
[2022-09-22T11:55:52,529][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [parent-join]
[2022-09-22T11:55:52,529][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [percolator]
[2022-09-22T11:55:52,530][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [rank-eval]
[2022-09-22T11:55:52,530][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [reindex]
[2022-09-22T11:55:52,531][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [repository-url]
[2022-09-22T11:55:52,531][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [systemd]
[2022-09-22T11:55:52,532][INFO ][o.e.p.PluginsService ] [Graylog] loaded module [transport-netty4]
[2022-09-22T11:55:52,533][INFO ][o.e.p.PluginsService ] [Graylog] no plugins loaded
[2022-09-22T11:55:52,604][INFO ][o.e.e.NodeEnvironment ] [Graylog] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [79.3gb], net total_space [97.8gb], types [ext4]
[2022-09-22T11:55:52,605][INFO ][o.e.e.NodeEnvironment ] [Graylog] heap size [8gb], compressed ordinary object pointers [true]
[2022-09-22T11:55:52,791][INFO ][o.e.n.Node ] [Graylog] node name [Graylog], node ID [k6eeD6KWQFuxleWoC0vZCw], cluster name [elasticsearch], roles [master, remote_cluster_client, data, ingest]
[2022-09-22T11:55:59,498][INFO ][o.e.t.NettyAllocator ] [Graylog] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-09-22T11:55:59,628][INFO ][o.e.d.DiscoveryModule ] [Graylog] using discovery type [zen] and seed hosts providers [settings]
[2022-09-22T11:56:00,138][WARN ][o.e.g.DanglingIndicesState] [Graylog] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-09-22T11:56:00,483][INFO ][o.e.n.Node ] [Graylog] initialized
[2022-09-22T11:56:00,484][INFO ][o.e.n.Node ] [Graylog] starting ...
[2022-09-22T11:56:00,706][INFO ][o.e.t.TransportService ] [Graylog] publish_address {172.16.64.203:9300}, bound_addresses {172.16.64.203:9300}
[2022-09-22T11:56:01,079][INFO ][o.e.b.BootstrapChecks ] [Graylog] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-09-22T11:56:01,083][INFO ][o.e.c.c.Coordinator ] [Graylog] cluster UUID [mOeSNqxhTlyN6sjIx8QmCw]
[2022-09-22T11:56:01,223][INFO ][o.e.c.s.MasterService ] [Graylog] elected-as-master ([1] nodes joined)[{Graylog}{k6eeD6KWQFuxleWoC0vZCw}{orTVpafwSAuUh5vFrxdUwg}{172.16.64.203}{172.16.64.203:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 16, version: 152, delta: master node changed {previous [], current [{Graylog}{k6eeD6KWQFuxleWoC0vZCw}{orTVpafwSAuUh5vFrxdUwg}{172.16.64.203}{172.16.64.203:9300}{dimr}]}
[2022-09-22T11:56:01,317][INFO ][o.e.c.s.ClusterApplierService] [Graylog] master node changed {previous [], current [{Graylog}{k6eeD6KWQFuxleWoC0vZCw}{orTVpafwSAuUh5vFrxdUwg}{172.16.64.203}{172.16.64.203:9300}{dimr}]}, term: 16, version: 152, reason: Publication{term=16, version=152}
[2022-09-22T11:56:01,352][INFO ][o.e.h.AbstractHttpServerTransport] [Graylog] publish_address {172.16.64.203:9200}, bound_addresses {172.16.64.203:9200}
[2022-09-22T11:56:01,352][INFO ][o.e.n.Node ] [Graylog] started
[2022-09-22T11:56:01,597][INFO ][o.e.g.GatewayService ] [Graylog] recovered [3] indices into cluster_state```
and /var/log/graylog-server/server.log
2022-09-22T12:05:47.969+03:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-09-22T12:05:49.293+03:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.3.7 [org.graylog.aws.AWSPlugin]
2022-09-22T12:05:49.295+03:00 INFO [CmdLineTool] Loaded plugin: Collector 4.3.7 [org.graylog.plugins.collector.CollectorPlugin]
2022-09-22T12:05:49.297+03:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.7 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-09-22T12:05:49.297+03:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.7+05bccc7 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-09-22T12:05:49.298+03:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.7+05bccc7 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-09-22T12:05:49.333+03:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-09-22T12:05:50.153+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2022-09-22T12:05:50.245+03:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-09-22T12:05:50.289+03:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:9}] to localhost:27017
2022-09-22T12:05:50.306+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=9932882}
2022-09-22T12:05:50.347+03:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:10}] to localhost:27017
2022-09-22T12:05:50.393+03:00 INFO [connection] Closed connection [connectionId{localValue:2, serverValue:10}] to localhost:27017 because the pool has been closed.
2022-09-22T12:05:50.396+03:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-09-22T12:05:50.554+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /172.16.64.203:9200. - Connection refused (Connection refused).
2022-09-22T12:05:50.556+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #1
2022-09-22T12:05:55.561+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /172.16.64.203:9200. - Connection refused (Connection refused).
2022-09-22T12:05:55.562+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #2
2022-09-22T12:06:00.567+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /172.16.64.203:9200. - Connection refused (Connection refused).
2022-09-22T12:06:00.568+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #3
2022-09-22T12:06:05.574+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /172.16.64.203:9200. - Connection refused (Connection refused).
2022-09-22T12:06:05.575+03:00 INFO [VersionProbe] Elasticsearch is not available. Retry #4