New Graylog 5 Open Deployment Inaccessible from web interface

Graylog Central (peer support)
1

1. Describe your incident:
Recently installed GrayPoint 5 on RHEL 8 box. Followed Graylog documentation to setup running both MongoDB 6.0.3 and Opensearch 2.3.0. I am running into an issue where I am unable to access the web interface both by hostname and by IP address. I’ve tried changing several configuration file settings to no avail. Attaching all relevant logs and config files below. Any help is appreciated.

From: sudo grep -v ^# /etc/graylog/server/server.conf

node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_email = xxx@cfpua.org
root_timezone=UTC
elasticsearch_discovery_zen_ping_unicast_hosts = xxx.xx.80.9:9300
elasticsearch_shards=1
script.inline: false
script.indexed: false
script.file: false
rest_listen_uri = http://xxx.xx.80.9:12900/
web_listen_uri = http://xxx.xx.80.9:9000/
root_password_sha2 = xxx
root_timezone = EST
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = xxx.xx.80.9:9000
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000```

```# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1
# Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.


#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:```

```# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog2
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: graylog2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
# http.port: 9200
#```

```[jake@graylog2 opensearch]$ netstat -an | grep 9000
tcp        0      0 xxx.xx.80.9:60752       xxx.xx.80.9:9000        TIME_WAIT  
tcp        0      0 xxx.xx.80.9:40070       xxx.xx.80.9:9000        TIME_WAIT  
tcp6       0      0 xxx.xx.80.9:9000        :::*                    LISTEN     
[jake@graylog2 opensearch]$ nc -vz graylog2.corp.cfpua.org 9000
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to xxxx::xxx:xxxx:fe8b:5ae failed: Connection refused.
Ncat: Trying next address...
Ncat: Connected to xxx.xx.80.9:9000.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[jake@graylog2 opensearch]$ nc -vz xxx.xx.80.9 9000            
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to xxx.xx.80.9:9000.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[jake@graylog2 opensearch]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:8b:05:ae brd ff:ff:ff:ff:ff:ff
    inet xxx.xx.80.9/28 brd xxx.xx.80.15 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet6 xxxx::xxx:xxxx:fe8b:5ae/64 scope link 
       valid_lft forever preferred_lft forever
[jake@graylog2 opensearch]$ sudo systemctl status graylog-server
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-01-19 13:03:34 EST; 35min ago
     Docs: http://docs.graylog.org/
 Main PID: 1505 (graylog-server)
    Tasks: 117 (limit: 102260)
   Memory: 1.0G
   CGroup: /system.slice/graylog-server.service
           ├─1505 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─1509 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Djava.io.tmpdir=/var/graylog/tmp -jar >

Jan 19 13:03:34 graylog2.corp.cfpua.org systemd[1]: Started Graylog server.
Jan 19 13:03:38 graylog2.corp.cfpua.org graylog-server[1509]: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

[jake@graylog2 opensearch]$ sudo systemctl status opensearch
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-01-19 13:04:21 EST; 34min ago
     Docs: https://opensearch.org/
 Main PID: 1162 (java)
    Tasks: 51 (limit: 102260)
   Memory: 1.3G
   CGroup: /system.slice/opensearch.service
           └─1162 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -D>

Jan 19 13:03:22 graylog2.corp.cfpua.org systemd[1]: Starting OpenSearch...
Jan 19 13:03:54 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: A terminally deprecated method in java.lang.System has been called
Jan 19 13:03:54 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.3.0.jar)
Jan 19 13:03:54 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jan 19 13:03:54 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: System::setSecurityManager will be removed in a future release
Jan 19 13:04:02 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: A terminally deprecated method in java.lang.System has been called
Jan 19 13:04:02 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.3.0.jar)
Jan 19 13:04:02 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jan 19 13:04:02 graylog2.corp.cfpua.org systemd-entrypoint[1162]: WARNING: System::setSecurityManager will be removed in a future release
Jan 19 13:04:21 graylog2.corp.cfpua.org systemd[1]: Started OpenSearch.

[jake@graylog2 opensearch]$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-01-19 13:03:43 EST; 35min ago
     Docs: https://docs.mongodb.org/manual
  Process: 1513 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS)
  Process: 1510 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 1506 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 1504 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
 Main PID: 1542 (mongod)
   Memory: 238.4M
   CGroup: /system.slice/mongod.service
           └─1542 /usr/bin/mongod -f /etc/mongod.conf

Jan 19 13:03:34 graylog2.corp.cfpua.org systemd[1]: Starting MongoDB Database Server...
Jan 19 13:03:35 graylog2.corp.cfpua.org mongod[1513]: about to fork child process, waiting until server is ready for connections.
Jan 19 13:03:35 graylog2.corp.cfpua.org mongod[1542]: forked process: 1542
Jan 19 13:03:43 graylog2.corp.cfpua.org mongod[1513]: child process started successfully, parent exiting
Jan 19 13:03:43 graylog2.corp.cfpua.org systemd[1]: Started MongoDB Database Server.```
2

That server.conf looks odd. We haven’t used rest_listen_uri or web_listen_uri for a long time. You should start with the version that comes from the v5 install.

3

I trimmed off some of the fat that I had added (think I read it from another blog post when I was troubleshooting). Restarted server with same results. See below server.conf.

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = 
root_password_sha2 = 
root_timezone = EST
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = xxx.xx.80.9:9000
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000```
4

I presume you omitted your password secret and root password, because not having that will definitely prevent it from starting.

Check your Graylog server logs. See if it has any clues.

https://go2docs.graylog.org/5-0/setting_up_graylog/default_file_locations.html?Highlight=default%20files

6

That is correct. Please see below server.log.

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-01-19T15:12:04.536-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-19T15:13:38.851-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[scripting_api_preview=off, search_filter=on]}
2023-01-19T15:13:44.645-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 5.0.2 [org.graylog.aws.AWSPlugin]
2023-01-19T15:13:44.653-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 5.0.2 [org.graylog.integrations.IntegrationsPlugin]
2023-01-19T15:13:44.664-05:00 INFO  [CmdLineTool] Loaded plugin: Collector 5.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2023-01-19T15:13:44.665-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 5.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2023-01-19T15:13:44.666-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 5.0.2+59d96f8 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2023-01-19T15:13:44.666-05:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 5.0.2+59d96f8 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2023-01-19T15:13:44.862-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Djava.io.tmpdir=/var/graylog/tmp -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2023-01-19T15:13:46.594-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7187bac9]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-19T15:13:46.634-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7187bac9]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-19T15:13:46.684-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:1}] to localhost:27017
2023-01-19T15:13:46.685-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:2}] to localhost:27017
2023-01-19T15:13:46.684-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=175746025}
2023-01-19T15:13:46.969-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2023-01-19T15:13:47.241-05:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.3
2023-01-19T15:13:50.806-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-01-19T15:13:50.811-05:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #1
2023-01-19T15:13:55.822-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-01-19T15:13:55.828-05:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #2
2023-01-19T15:14:00.833-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-01-19T15:14:00.834-05:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #3
2023-01-19T15:14:05.835-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-01-19T15:14:05.836-05:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #4
2023-01-19T15:14:10.837-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2023-01-19T15:14:10.837-05:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #5
2023-01-19T15:14:16.353-05:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:2.3.0>
2023-01-19T15:14:16.749-05:00 INFO  [Version] HV000001: Hibernate Validator null
2023-01-19T15:14:21.503-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2023-01-19T15:14:21.537-05:00 INFO  [NodeId] Node ID: 372c7f7d-de5d-4818-8528-8b927c312b81
2023-01-19T15:14:21.867-05:00 INFO  [LogManager] Loading logs.
2023-01-19T15:14:21.897-05:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2023-01-19T15:14:21.951-05:00 INFO  [LogManager] Logs loading complete.
2023-01-19T15:14:21.954-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2023-01-19T15:14:21.974-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7187bac9]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-19T15:14:21.977-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2023-01-19T15:14:21.979-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=9171918}
2023-01-19T15:14:21.980-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017
2023-01-19T15:14:21.981-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7187bac9]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-19T15:14:21.982-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-19T15:14:21.986-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017
2023-01-19T15:14:22.304-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2023-01-19T15:14:22.965-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running OpenSearch:2.3.0
2023-01-19T15:14:24.161-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2023-01-19T15:14:24.351-05:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017
2023-01-19T15:14:24.375-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2023-01-19T15:14:24.940-05:00 WARN  [PlatformDependent] Failed to get the temporary directory; falling back to: /tmp
2023-01-19T15:14:25.262-05:00 INFO  [ServerBootstrap] Graylog server 5.0.2+59d96f8 starting up
2023-01-19T15:14:25.262-05:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.5 on Linux 4.18.0-372.32.1.el8_6.x86_64
2023-01-19T15:14:25.262-05:00 INFO  [ServerBootstrap] Deployment: rpm
2023-01-19T15:14:25.262-05:00 INFO  [ServerBootstrap] OS: Red Hat Enterprise Linux 8.4 (Ootpa) (rhel)
2023-01-19T15:14:25.263-05:00 INFO  [ServerBootstrap] Arch: amd64
2023-01-19T15:14:25.471-05:00 INFO  [ServerBootstrap] Running 49 migrations...
2023-01-19T15:14:26.305-05:00 INFO  [PeriodicalsService] Starting 26 periodicals ...
2023-01-19T15:14:26.305-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.400-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2023-01-19T15:14:26.418-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.538-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2023-01-19T15:14:26.550-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2023-01-19T15:14:26.552-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2023-01-19T15:14:26.552-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2023-01-19T15:14:26.559-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2023-01-19T15:14:26.568-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s].
2023-01-19T15:14:26.573-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2023-01-19T15:14:26.577-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2023-01-19T15:14:26.597-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.630-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2023-01-19T15:14:26.638-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2023-01-19T15:14:26.641-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.677-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2023-01-19T15:14:26.688-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2023-01-19T15:14:26.693-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.698-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s].
2023-01-19T15:14:26.728-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2023-01-19T15:14:26.744-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2023-01-19T15:14:26.751-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever.
2023-01-19T15:14:26.763-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2023-01-19T15:14:26.770-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2023-01-19T15:14:26.771-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2023-01-19T15:14:26.790-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2023-01-19T15:14:26.790-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2023-01-19T15:14:26.897-05:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:8}] to localhost:27017
2023-01-19T15:14:26.899-05:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017
2023-01-19T15:14:29.247-05:00 WARN  [MultiPartReaderClientSide] Cannot create temporary files. Multipart attachments will be limited to "4,096" bytes.
java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?]
        at java.io.File.createTempFile(Unknown Source) ~[?:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.createMimeConfig(MultiPartReaderClientSide.java:119) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.<init>(MultiPartReaderClientSide.java:101) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide.<init>(MultiPartReaderServerSide.java:57) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
        at java.util.LinkedList$LLSpliterator.forEachRemaining(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:307) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getCustomProviders(Providers.java:151) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory.initialize(MessageBodyFactory.java:219) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory$MessageBodyWorkersConfigurator.postInit(MessageBodyFactory.java:114) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$2(ApplicationHandler.java:353) ~[graylog.jar:?]
        at java.util.Arrays$ArrayList.forEach(Unknown Source) ~[?:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:353) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:297) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:296) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:261) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:248) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.<init>(GrizzlyHttpContainer.java:310) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:163) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.setUp(JerseyService.java:314) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:194) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:158) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-01-19T15:14:30.579-05:00 INFO  [JerseyService] Started REST API at <172.31.80.9:9000>
2023-01-19T15:14:30.579-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=1, FailureHandlingService [RUNNING]=2, GracefulShutdownService [RUNNING]=3, UserSessionTerminationService [RUNNING]=3, BufferSynchronizerService [RUNNING]=4, UrlWhitelistService [RUNNING]=7, PrometheusExporter [RUNNING]=7, OutputSetupService [RUNNING]=8, LocalKafkaMessageQueueWriter [RUNNING]=10, EtagService [RUNNING]=12, LocalKafkaMessageQueueReader [RUNNING]=15, StreamCacheService [RUNNING]=41, LocalKafkaJournal [RUNNING]=53, LookupTableService [RUNNING]=56, JobSchedulerService [RUNNING]=57, MongoDBProcessingStatusRecorderService [RUNNING]=109, ConfigurationEtagService [RUNNING]=113, GeoIpDbFileChangeMonitorService [RUNNING]=130, PeriodicalsService [RUNNING]=492, JerseyService [RUNNING]=4274}
2023-01-19T15:14:30.583-05:00 INFO  [ServiceManagerListener] Services are healthy
2023-01-19T15:14:30.594-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2023-01-19T15:14:30.611-05:00 INFO  [ServerBootstrap] Graylog server up and running.```
7

I posted some log files but they are being blocked by the spam filter. Yeah, I removed the password secret and root password. I guess when the filter processes the logs you will see the error I am getting.

8

Hey @lawdfarquhar

I have to agree with @joe.gross , your configuration file is not correct for the version your using.

This is from Graylog 2.4

elasticsearch_discovery_zen_ping_unicast_hosts = xxx.xx.80.9:9300
elasticsearch_shards=1
script.inline: false
script.indexed: false
script.file: false
rest_listen_uri = http://xxx.xx.80.9:12900/
web_listen_uri = http://xxx.xx.80.9:9000/

Im not sure where you got this conig, dont see it in the default config file.

stream_aware_field_types=false

Please look at the default graylog configuration file here.

This is the basic setup here

1 Like
9

Yeah that first block you posted I think I pulled from my 2.4 instance I have up and running on the network when I was troubleshooting originally but it has already been removed. I’m starting to wonder if it is even possible to have two single node instance running at the same time (the old one is live and in production but is on the old version and this one will replace it but they have different IPs). Also, my server.log cleared the spam filter so it is posted above. I think the stream aware is part of the default config. I did not touch that part. It looks like this in my config file:

# If set to "true", Graylog will periodically investigate indices to figure out which fields are used in which streams.
# It will make field list in Graylog interface show only fields used in selected streams, but can decrease system performance,
# especially on systems with great number of streams and fields.
stream_aware_field_types=false
10

I corrected the graylog server.conf file and am still not seeing the front end of graylog. Does anyone have any ideas on this?

[jake@graylog2 ~]$ cat /etc/graylog/server/server.conf         | egrep -v "^\s*(#|$)"
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = EST
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = xxx.xx.80.9:9000
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000

[jake@graylog2 ~]$ sudo systemctl restart graylog-server
[jake@graylog2 ~]$ tail -f /var/log/graylog-server/server.log
2023-01-23T09:38:12.250-05:00 INFO  [FailureHandlingService] The service's thread has been interrupted. The queue currently contains 0 failure batches.
2023-01-23T09:38:12.250-05:00 INFO  [FailureSubmissionQueue] [FailureHandlerService#triggerShutdown] Total number of submitted batches: 0 (0 failures), total number of consumed batches: 0 (0 failures)
2023-01-23T09:38:12.256-05:00 INFO  [JerseyService] Shutting down HTTP listener at <xxx.xx.80.9:9000>
2023-01-23T09:38:12.262-05:00 INFO  [LocalKafkaMessageQueueReader] Stopping.
2023-01-23T09:38:12.266-05:00 INFO  [LogManager] Shutting down.
2023-01-23T09:38:12.293-05:00 INFO  [LogManager] Shutdown complete.
2023-01-23T09:38:12.309-05:00 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2023-01-23T09:38:15.250-05:00 INFO  [FailureHandlingService] Shutting down the service. Processed 0 remaining failure batches.
2023-01-23T09:38:15.251-05:00 INFO  [FailureSubmissionQueue] [FailureHandlerService#shutDown] Total number of submitted batches: 0 (0 failures), total number of consumed batches: 0 (0 failures)
2023-01-23T09:38:15.251-05:00 INFO  [ServiceManagerListener] Services are now stopped.
2023-01-23T09:38:18.251-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[scripting_api_preview=off, search_filter=on]}
2023-01-23T09:38:19.513-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 5.0.2 [org.graylog.aws.AWSPlugin]
2023-01-23T09:38:19.516-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 5.0.2 [org.graylog.integrations.IntegrationsPlugin]
2023-01-23T09:38:19.517-05:00 INFO  [CmdLineTool] Loaded plugin: Collector 5.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2023-01-23T09:38:19.521-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 5.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2023-01-23T09:38:19.522-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 5.0.2+59d96f8 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2023-01-23T09:38:19.522-05:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 5.0.2+59d96f8 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2023-01-23T09:38:19.567-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Djava.io.tmpdir=/var/graylog/tmp -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2023-01-23T09:38:19.997-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5a2bd7c8]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-23T09:38:20.010-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5a2bd7c8]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-23T09:38:20.022-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:12}] to localhost:27017
2023-01-23T09:38:20.024-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=45641539}
2023-01-23T09:38:20.022-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:11}] to localhost:27017
2023-01-23T09:38:20.091-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:13}] to localhost:27017
2023-01-23T09:38:20.177-05:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.3
2023-01-23T09:38:21.252-05:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:2.3.0>
2023-01-23T09:38:21.502-05:00 INFO  [Version] HV000001: Hibernate Validator null
2023-01-23T09:38:25.961-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2023-01-23T09:38:25.988-05:00 INFO  [NodeId] Node ID: 372c7f7d-de5d-4818-8528-8b927c312b81
2023-01-23T09:38:26.273-05:00 INFO  [LogManager] Loading logs.
2023-01-23T09:38:26.346-05:00 INFO  [LogManager] Logs loading complete.
2023-01-23T09:38:26.350-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2023-01-23T09:38:26.371-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5a2bd7c8]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-23T09:38:26.375-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:14}] to localhost:27017
2023-01-23T09:38:26.377-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6593634}
2023-01-23T09:38:26.379-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:15}] to localhost:27017
2023-01-23T09:38:26.380-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-372.32.1.el8_6.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5a2bd7c8]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2023-01-23T09:38:26.382-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-23T09:38:26.388-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:16}] to localhost:27017
2023-01-23T09:38:26.668-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2023-01-23T09:38:27.351-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running OpenSearch:2.3.0
2023-01-23T09:38:28.491-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2023-01-23T09:38:28.668-05:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:17}] to localhost:27017
2023-01-23T09:38:28.695-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2023-01-23T09:38:29.277-05:00 WARN  [PlatformDependent] Failed to get the temporary directory; falling back to: /tmp
2023-01-23T09:38:29.634-05:00 INFO  [ServerBootstrap] Graylog server 5.0.2+59d96f8 starting up
2023-01-23T09:38:29.635-05:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.5 on Linux 4.18.0-372.32.1.el8_6.x86_64
2023-01-23T09:38:29.639-05:00 INFO  [ServerBootstrap] Deployment: rpm
2023-01-23T09:38:29.640-05:00 INFO  [ServerBootstrap] OS: Red Hat Enterprise Linux 8.4 (Ootpa) (rhel)
2023-01-23T09:38:29.641-05:00 INFO  [ServerBootstrap] Arch: amd64
2023-01-23T09:38:29.879-05:00 INFO  [ServerBootstrap] Running 49 migrations...
2023-01-23T09:38:30.746-05:00 INFO  [PeriodicalsService] Starting 26 periodicals ...
2023-01-23T09:38:30.748-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2023-01-23T09:38:30.864-05:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:18}] to localhost:27017
2023-01-23T09:38:30.867-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2023-01-23T09:38:30.888-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2023-01-23T09:38:30.893-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2023-01-23T09:38:30.898-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2023-01-23T09:38:30.902-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2023-01-23T09:38:30.903-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2023-01-23T09:38:30.905-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2023-01-23T09:38:30.909-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s].
2023-01-23T09:38:31.037-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2023-01-23T09:38:31.040-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2023-01-23T09:38:31.044-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2023-01-23T09:38:31.052-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2023-01-23T09:38:31.052-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2023-01-23T09:38:31.054-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2023-01-23T09:38:31.057-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2023-01-23T09:38:31.060-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2023-01-23T09:38:31.062-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2023-01-23T09:38:31.063-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s].
2023-01-23T09:38:31.064-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2023-01-23T09:38:31.082-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2023-01-23T09:38:31.089-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever.
2023-01-23T09:38:31.098-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2023-01-23T09:38:31.112-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2023-01-23T09:38:31.113-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2023-01-23T09:38:31.115-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2023-01-23T09:38:31.116-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2023-01-23T09:38:33.288-05:00 WARN  [MultiPartReaderClientSide] Cannot create temporary files. Multipart attachments will be limited to "4,096" bytes.
java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?]
        at java.io.File.createTempFile(Unknown Source) ~[?:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.createMimeConfig(MultiPartReaderClientSide.java:119) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.<init>(MultiPartReaderClientSide.java:101) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide.<init>(MultiPartReaderServerSide.java:57) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
        at java.util.LinkedList$LLSpliterator.forEachRemaining(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:307) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getCustomProviders(Providers.java:151) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory.initialize(MessageBodyFactory.java:219) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory$MessageBodyWorkersConfigurator.postInit(MessageBodyFactory.java:114) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$2(ApplicationHandler.java:353) ~[graylog.jar:?]
        at java.util.Arrays$ArrayList.forEach(Unknown Source) ~[?:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:353) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:297) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:296) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:261) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:248) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.<init>(GrizzlyHttpContainer.java:310) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:163) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.setUp(JerseyService.java:314) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:194) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:158) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-01-23T09:38:34.636-05:00 INFO  [JerseyService] Started REST API at <xxx.xx.80.9:9000>
2023-01-23T09:38:34.638-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {LocalKafkaMessageQueueWriter [RUNNING]=1, OutputSetupService [RUNNING]=2, FailureHandlingService [RUNNING]=2, UserSessionTerminationService [RUNNING]=3, BufferSynchronizerService [RUNNING]=4, PrometheusExporter [RUNNING]=4, InputSetupService [RUNNING]=4, GracefulShutdownService [RUNNING]=7, LocalKafkaMessageQueueReader [RUNNING]=9, UrlWhitelistService [RUNNING]=11, EtagService [RUNNING]=11, GeoIpDbFileChangeMonitorService [RUNNING]=14, ConfigurationEtagService [RUNNING]=32, JobSchedulerService [RUNNING]=41, LocalKafkaJournal [RUNNING]=82, StreamCacheService [RUNNING]=98, LookupTableService [RUNNING]=100, MongoDBProcessingStatusRecorderService [RUNNING]=108, PeriodicalsService [RUNNING]=373, JerseyService [RUNNING]=3892}
2023-01-23T09:38:34.642-05:00 INFO  [ServiceManagerListener] Services are healthy
2023-01-23T09:38:34.644-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2023-01-23T09:38:34.655-05:00 INFO  [ServerBootstrap] Graylog server up and running.
11

Hey @lawdfarquhar

and again Im not finding this setting in graylog configuration file.

image
image1881×375 29.4 KB

What I got out of the logs were about java

WARN  [PlatformDependent] Failed to get the temporary directory; falling back to: /tmp

java.io.IOException: Permission denied

But I also see this

INFO  [ServiceManagerListener] Services are healthy

Permissions set on Graylog’s directory? and are you sure your using Graylog 5.0 configuration file?

did you check firewalls, selinux, etc…?

1 Like
12

@gsmith here is the article referencing stream_aware_field_types=false

https://go2docs.graylog.org/5-0/making_sense_of_your_log_data/streams.html

I’m not sure why it isn’t showing up in the graylog config page on the site.

On another note, which I’m wondering if this might be the problem. I have an older graylog server that is up and running on the same network (.10 instead of .9). They are both single node instances and this one (graylog 5) is meant to replace it. Since they are both set to master in the server.conf file, could this be the problem? Do I need to change the name of this server to just graylog instead of graylog2?

Here are the perms and firewall configs. I have selinux disabled at the moment.

[jake@graylog2 graylog]$ ls -la
total 12
drwxr-xr-x.  3 root root   20 Jan 12 10:24 .
drwxr-xr-x. 94 root root 8192 Jan 19 15:13 ..
drwxr-xr-x.  2 root root   58 Jan 18 12:06 server
[jake@graylog2 graylog]$ cd server/
[jake@graylog2 server]$ ls -la
total 44
drwxr-xr-x. 2 root root    58 Jan 18 12:06 .
drwxr-xr-x. 3 root root    20 Jan 12 10:24 ..
-rw-r--r--. 1 root root  1930 Jan  4 09:58 log4j2.xml
-rw-r--r--. 1 root root    37 Jan 12 10:24 node-id
-rw-r--r--. 1 root root 36851 Jan 19 15:09 server.conf


[jake@graylog2 server]$ sudo firewall-cmd --state
running
[jake@graylog2 server]$ sudo firewall-cmd --get-default-zone
public
[jake@graylog2 server]$ sudo firewall-cmd --get-active-zone
public
  interfaces: ens192
[jake@graylog2 server]$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
13

Hey

Oh i see now, yeah that should be present in the file config.

If there not in a cluster it should make a difference.

For testing purposes you could try disabling the firewall and see if that would help, if not enable it back.

So root owns Graylog’s directory? Is graylog a root user? The reason ai ask this is because the logs show permission issue.

By chance do you have a nginx or apache installed, meaning reverse proxy configured?
If all you service are running

  • systemctl status graylog-server
  • systemctl status mongod
  • systemctl status opensearch/elasticsearch

And your unable to login on the GUI, then that tells me something is blocking your connection.

EDIT:

I over looked this, Correct me if im wrong but you have 2 servers with the same FQDN? if so then yeah that might be an issue. Thats kind of like have two servers with the same IP Address there will be issues on the network for sure.

14

I tried this but it didn’t seem to change much. Still no access to the webpage.

Yes and I noticed on my working graylog server that it is the same way permissions wise. I didn’t change any permissions on the graylog account.

Yeah I noticed that it can’t seem to make a temp directory for something java related.

2023-01-25T08:09:45.402-05:00 WARN  [MultiPartReaderClientSide] Cannot create temporary files. Multipart attachments will be limited to "4,096" bytes.
java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[?:?]
        at java.io.File.createTempFile(Unknown Source) ~[?:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.createMimeConfig(MultiPartReaderClientSide.java:119) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.<init>(MultiPartReaderClientSide.java:101) ~[graylog.jar:?]
        at org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide.<init>(MultiPartReaderServerSide.java:57) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:59) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:47) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:74) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
        at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:131) ~[graylog.jar:?]
        at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:176) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:98) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93) ~[graylog.jar:?]
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
        at java.util.LinkedList$LLSpliterator.forEachRemaining(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140) ~[graylog.jar:?]
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:307) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.inject.Providers.getCustomProviders(Providers.java:151) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory.initialize(MessageBodyFactory.java:219) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory$MessageBodyWorkersConfigurator.postInit(MessageBodyFactory.java:114) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$2(ApplicationHandler.java:353) ~[graylog.jar:?]
        at java.util.Arrays$ArrayList.forEach(Unknown Source) ~[?:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:353) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:297) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:296) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:261) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:248) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.<init>(GrizzlyHttpContainer.java:310) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory.createHttpServer(GrizzlyHttpServerFactory.java:163) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.setUp(JerseyService.java:314) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:194) [graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:158) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

Negative

All active and enabled.

I was really thinking this was the firewall config because I noticed a different config on my working server that used IPsets as sources in the trusted zone and defaulted to drop.

No, sorry for the confusion. I have two servers with two names. One is graylog with an IP of .10 which is up and running on an older version of Graylog and the other is graylog2 with an IP of .9 which is not running (at least the webpage isn’t).

I also compared my iproutes against my working graylog and they are the same, default routes. I’m wondering if those IPsets in the trusted zone of the firewall were kind of acting as routes. I will admit that I don’t know very much about firewalld so any insight is appreciated.

The below is from the working graylog server not the one in question for the sake of clarity

[jp@graylog](357)$ sudo firewall-cmd --permanent --info-ipset=admin-ips
[sudo] password for jp: 
admin-ips
  type: hash:ip
  options: 
  entries: xxx.xx.69.0/28 xxx.xx.70.0/28 xxx.xx.80.224/28
[jp@graylog](358)$ sudo firewall-cmd --get-active-zone                 
work
  interfaces: ens192
trusted
  interfaces: lo
  sources: ipset:admin-ips
15

Hello

I’m in the same boat, but i get by. I use Iptables with some other stuff, mainly because Ive been using it for 17 + years.

Another thing i also caught was the servers at .10 you can logon but the .9 network you can not by chance the web UI your using what network is that on? Also can you ping the .9 network from from the device with you Web UI?

16

Correct. To the web interface.

They are both in the same subnet. xxx.xx.80.0/28

I can ping .10 from .9 and vice versa. I can also ping them from my workstation.

17

hey,
Ok so I went back over the logs again, Im wondering if you could post these command here.

[root@graylog server]# mongod -version

And check Opensearch, I think your using loop back address 127.0.0.1 so it would be something like this

curl -XGET http://127.0.0.1:9200/

EDIT: I forgot to ask , when you use IP address to logon for this Graylog node what does the Browser show?
Not usre if this will help but here is my opensearch config file

[root@graylog server]#  cat /etc/opensearch/opensearch.yml  | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false
discovery.type: single-node
bootstrap.memory_lock: true
plugins.security.disabled: true
plugins.security.system_indices.enabled: false
18 
[jake@graylog2 ~]$ mongod --version
db version v6.0.3
Build Info: {
    "version": "6.0.3",
    "gitVersion": "f803681c3ae19817d31958965850193de067c516",
    "openSSLVersion": "OpenSSL 1.1.1g FIPS  21 Apr 2020",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "rhel80",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

[jake@graylog2 ~]$ curl -XGET http://127.0.0.1:9200/
{
  "name" : "graylog2",
  "cluster_name" : "graylog2",
  "cluster_uuid" : "_RtygMwwTza-sul_cPnz4g",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.3.0",
    "build_type" : "rpm",
    "build_hash" : "6f6e84ebc54af31a976f53af36a5c69d474a5140",
    "build_date" : "2022-09-09T00:07:32.109283578Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"

IDK why but when I tried this this morning I was able to get in (xxx.xx.80.9:9000). Using the full name didn’t work but at least I can hit the web interface now. I just turned the firewall back on too so I think i’ve kind of ruled that out. Very spooky and I’m a little confused now.

cluster.name: graylog2 
node.name: graylog2
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0 
discovery.type: single-node
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
action.auto_create_index: false
plugins.security.disabled: true
19

Hey @lawdfarquhar

That must be the dark magic of Graylog :laughing:

I am curious what happen and how your relsolving this issue.

I belive that would be a DNS issue, you can check by ping’ing <FQDN>.something on a different node.

20

My best bet would be taking the firewall out of public zone but not sure.

The A record is in DNS. I think the problem is that Graylog uses port 9000 and DNS only cares about 80 and 443. I need to set it up to run on 443. Does Graylog recommend changing the default port to 443 in server.conf or is their some other way of making this work without using proxies.

Thanks for all the help.

21

Hey @lawdfarquhar

Correction, DNS is port 53

Port 80/443 is for HTTP/HTTPS