1. Describe your incident:
Graylog starts and connects to OpenSearch successfully, but does not bind to port 9000 - netstat -tuln | grep :9000 returns nothing. UFW is inactive on the server and I explicitly allowed port 9000 in iptables.
The intention is to proxy through Apache 2 (I have tested the proxy configuration and it is working), but at this point, just getting it to work locally would be nice.
MongoDB, OpenSearch and graylog-server installed successfully according to instructions at Ubuntu Installation. OpenSearch binds to ports 9200 and 9300 successfully.
2. Describe your environment:
- OS Information:
Ubuntu 20.04.4
- Package Version:
graylog-server 5.2.6-1
opensearch 2.13.0
mongodb-org 6.0.15
- Service logs, configurations, and environment variables:
/etc/graylog/server/server.conf differences with default configuration:
password_secret = [redacted]
root_password_sha2 = [redacted]
http_bind_address = 127.0.0.1:9000
http_publish_uri = http://127.0.0.1:9000/
http_external_uri = http://127.0.0.1:9000/
systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-04-25 07:20:11 UTC; 16min ago
Docs: http://docs.graylog.org/
Main PID: 2731 (graylog-server)
Tasks: 26 (limit: 1049)
Memory: 265.8M
CGroup: /system.slice/graylog-server.service
├─2731 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─2732 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=deb /usr/share/graylog-server/graylog.jar server -f /etc/graylog/server/server.conf -np
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1997)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1862)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:180)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.featureflag.ImmutableFeatureFlagsCollector.logUsedFeatureFlags(ImmutableFeatureFlagsCollector.java:81)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.featureflag.ImmutableFeatureFlagsCollector.toMap(ImmutableFeatureFlagsCollector.java:66)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.featureflag.FeatureFlagsFactory.createImmutableFeatureFlags(FeatureFlagsFactory.java:38)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.featureflag.FeatureFlagsFactory.createImmutableFeatureFlags(FeatureFlagsFactory.java:26)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.bootstrap.CmdLineTool.getFeatureFlags(CmdLineTool.java:438)
Apr 25 07:35:26 host.domain graylog-server[2732]: at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:276)
Apr 25 07:35:26 host.domain graylog-server[2732]: ... 2 more
tail -f /var/log/graylog-server/server.log
2024-04-25T10:14:38.306Z INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[frontend_hotkeys=on, field_types_management=on, cloud_inputs=on, scripting_api_preview=on, composable_index_templates=off, search_filter=on, preflight_web=on, instant_archiving=off]}
2024-04-25T10:14:40.438Z INFO [CmdLineTool] Loaded plugin: AWS plugins 5.2.6+5296b15 [org.graylog.aws.AWSPlugin]
2024-04-25T10:14:40.472Z INFO [CmdLineTool] Loaded plugin: Integrations 5.2.6+5296b15 [org.graylog.integrations.IntegrationsPlugin]
2024-04-25T10:14:40.502Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 5.2.6+5296b15 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2024-04-25T10:14:40.527Z INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 5.2.6+5296b15 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2024-04-25T10:14:40.531Z INFO [CmdLineTool] Loaded plugin: OpenSearch 2 Support 5.2.6+5296b15 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2024-04-25T10:14:40.608Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=deb
2024-04-25T10:14:41.178Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.4.0-177-generic"}, "platform": "Java/Eclipse Adoptium/17.0.10+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5633ed82]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2024-04-25T10:14:41.192Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.4.0-177-generic"}, "platform": "Java/Eclipse Adoptium/17.0.10+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5633ed82]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2024-04-25T10:14:41.223Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=83757982}
2024-04-25T10:14:41.515Z INFO [MongoDBPreflightCheck] Connected to MongoDB version 6.0.15
2024-04-25T10:14:43.133Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.4.0-177-generic"}, "platform": "Java/Eclipse Adoptium/17.0.10+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5633ed82]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2024-04-25T10:14:43.136Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5695233}
2024-04-25T10:14:43.141Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.8.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.4.0-177-generic"}, "platform": "Java/Eclipse Adoptium/17.0.10+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@5633ed82]}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2024-04-25T10:14:43.143Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2024-04-25T10:14:43.182Z INFO [IndexerDiscoveryProvider] No indexer hosts configured, using fallback http://127.0.0.1:9200
2024-04-25T10:14:43.914Z INFO [FilePersistedNodeIdProvider] Node ID: 559faf75-988c-4668-aeef-d48fc374f754
2024-04-25T10:14:45.071Z INFO [IndexerDiscoveryProvider] No indexer hosts configured, using fallback http://127.0.0.1:9200
2024-04-25T10:14:45.109Z INFO [FilePersistedNodeIdProvider] Node ID: 559faf75-988c-4668-aeef-d48fc374f754
2024-04-25T10:14:47.278Z INFO [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:2.13.0>
2024-04-25T10:14:47.881Z INFO [Version] HV000001: Hibernate Validator null
2024-04-25T10:14:57.175Z INFO [InputBufferImpl] Message journal is enabled.
2024-04-25T10:14:57.238Z INFO [FilePersistedNodeIdProvider] Node ID: 559faf75-988c-4668-aeef-d48fc374f754
2024-04-25T10:14:57.813Z INFO [LogManager] Loading logs.
2024-04-25T10:14:57.936Z WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2024-04-25T10:14:58.048Z INFO [LogManager] Logs loading complete.
2024-04-25T10:14:58.063Z INFO [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2024-04-25T10:14:58.428Z INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
3. What steps have you already taken to try and solve the problem?
I have tried the solutions described at Graylog not listening on port 9000 - /tmp noexec issue and also tried setting the environment variable in /lib/systemd/system/graylog-server.service, replacing the ExecStart line with…
ExecStart=/bin/sh -c 'GRAYLOG_SERVER_JAVA_OPTS="$${GRAYLOG_SERVER_JAVA_OPTS} -Djava.io.tmpdir=/var/graylog/tmp" exec /usr/share/graylog-server/bin/graylog-server'
4. How can the community help?
Any hint as to why the server web UI is not starting properly / binding on the network would be much appreciated.