Graylog startup failed - Java IllegalStateException

hcomtes · September 27, 2021, 5:33pm

Description of your problem

Graylog no longer starts correctly. When I restart the application I get the following error from server.log in /var/log/graylog-server/:

2021-09-27T17:16:04.349Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:166) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:277) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
	Caused by: java.net.SocketException: Permission denied
		at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_292]
		at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_292]
		at sun.nio.ch.Net.bind(Net.java:453) ~[?:1.8.0_292]
		at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) ~[?:1.8.0_292]
		at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) ~[?:1.8.0_292]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:107) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:64) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:215) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:195) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:186) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:711) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:256) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:202) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:156) ~[graylog.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]

Description of steps you’ve taken to attempt to solve the issue

I’ve noticed that if I comment out the http_bind_address line from server.conf, the error does not appear and the server launches OK. However, I am unable to access the application.

My question is: why does this error occur if I have a http_bind_address set, and how can I fix it?

Environmental information

Ubuntu 20.04 LTS

Package versions

Graylog: 4.1.5
MongoDB: 1:3.6.9+really3.6.8+90~g8e540c0b6d-0ubuntu5.2
Elasticsearch: 7.10.2

hcomtes · September 27, 2021, 5:37pm

I would attach both server.conf and server.log but they exceed the character limit on this forum, and I can’t attach them to be uploaded due to your restrictions on file types. Let me know if you need them, and if so, how best to share them.

tmacgbay · September 27, 2021, 5:57pm

Try this:
$ cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Looks like it is a permission denied on a socket exception - at least that’s what I read out of it. Maybe you have some other service/application using the Graylog port?

hcomtes · September 29, 2021, 9:15am

Thanks for your response. Below is the server.conf file with password hashes removed:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = 
root_password_sha2 = 
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.1.3.163:80
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

I’m not aware of any application using port 80 on this machine, but I suppose it is possible. I used sudo ss | grep :80 to check port 80 and it returned nothing. Is there anything else I can use to check?

tmacgbay · September 29, 2021, 10:11am

Just to get things running, try binding to a different port, Graylog default is 9000…

http_bind_address = 10.1.3.163:9000

If that works fine then it’s just figuring what else you have going at :80

hcomtes · September 29, 2021, 2:20pm

When using port 9000, the error message doesn’t appear, and the server appears the start correctly:

2021-09-29T14:15:29.187Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.1.5 [org.graylog.aws.AWSPlugin]
2021-09-29T14:15:29.189Z INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.1.5 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-09-29T14:15:29.190Z INFO  [CmdLineTool] Loaded plugin: Integrations 4.1.5 [org.graylog.integrations.IntegrationsPlugin]
2021-09-29T14:15:29.191Z INFO  [CmdLineTool] Loaded plugin: Collector 4.1.5 [org.graylog.plugins.collector.CollectorPlugin]
2021-09-29T14:15:29.192Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.1.5 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-09-29T14:15:29.192Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.1.5 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2021-09-29T14:15:29.193Z INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.1.5 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2021-09-29T14:15:29.193Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.1.5 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-09-29T14:15:29.194Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.1.5+01c9198 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-09-29T14:15:29.194Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.1.5+01c9198 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-09-29T14:15:29.217Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-09-29T14:15:29.328Z INFO  [Version] HV000001: Hibernate Validator null
2021-09-29T14:15:31.778Z INFO  [InputBufferImpl] Message journal is enabled.
2021-09-29T14:15:31.794Z INFO  [NodeId] Node ID: 086ea892-b443-4092-aff8-ca9e3cdf0f36
2021-09-29T14:15:31.958Z INFO  [LogManager] Loading logs.
2021-09-29T14:15:32.104Z INFO  [LogManager] Logs loading complete.
2021-09-29T14:15:32.107Z INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-09-29T14:15:32.130Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-09-29T14:15:32.162Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-09-29T14:15:32.227Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:42260}] to localhost:27017
2021-09-29T14:15:32.236Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=7186517}
2021-09-29T14:15:32.246Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:42261}] to localhost:27017
2021-09-29T14:15:32.415Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-09-29T14:15:32.803Z INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-09-29T14:15:33.350Z INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-09-29T14:15:33.410Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-09-29T14:15:33.416Z INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-09-29T14:15:33.427Z INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:42262}] to localhost:27017
2021-09-29T14:15:33.446Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-09-29T14:15:33.486Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-09-29T14:15:33.510Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-09-29T14:15:33.548Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2021-09-29T14:15:34.208Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-09-29T14:15:34.210Z INFO  [ServerBootstrap] Graylog server 4.1.5+01c9198 starting up
2021-09-29T14:15:34.210Z INFO  [ServerBootstrap] JRE: Private Build 1.8.0_292 on Linux 5.4.0-86-generic
2021-09-29T14:15:34.210Z INFO  [ServerBootstrap] Deployment: deb
2021-09-29T14:15:34.210Z INFO  [ServerBootstrap] OS: Ubuntu 20.04.3 LTS (focal)
2021-09-29T14:15:34.210Z INFO  [ServerBootstrap] Arch: amd64
2021-09-29T14:15:34.238Z INFO  [PeriodicalsService] Starting 36 periodicals ...
2021-09-29T14:15:34.239Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-09-29T14:15:34.255Z INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-09-29T14:15:34.321Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-09-29T14:15:34.321Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-09-29T14:15:34.323Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-09-29T14:15:34.326Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-09-29T14:15:34.326Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-09-29T14:15:34.329Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-09-29T14:15:34.331Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-09-29T14:15:34.331Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-09-29T14:15:34.331Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-09-29T14:15:34.332Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-09-29T14:15:34.337Z INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-09-29T14:15:34.337Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-09-29T14:15:34.342Z INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:42264}] to localhost:27017
2021-09-29T14:15:34.344Z INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:42263}] to localhost:27017
2021-09-29T14:15:34.351Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-09-29T14:15:34.351Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-09-29T14:15:34.353Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-09-29T14:15:34.368Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-09-29T14:15:34.371Z INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:42265}] to localhost:27017
2021-09-29T14:15:34.374Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-09-29T14:15:34.389Z INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:42267}] to localhost:27017
2021-09-29T14:15:34.391Z INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:42266}] to localhost:27017
2021-09-29T14:15:34.404Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-09-29T14:15:34.404Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-09-29T14:15:34.416Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2021-09-29T14:15:34.416Z INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-09-29T14:15:34.425Z INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-09-29T14:15:34.434Z INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-09-29T14:15:34.442Z INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-09-29T14:15:34.449Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-09-29T14:15:34.454Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-09-29T14:15:34.468Z INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-09-29T14:15:34.470Z INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-09-29T14:15:34.471Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-09-29T14:15:34.472Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2021-09-29T14:15:34.475Z INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2021-09-29T14:15:34.475Z INFO  [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2021-09-29T14:15:34.475Z INFO  [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2021-09-29T14:15:34.476Z INFO  [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2021-09-29T14:15:34.476Z INFO  [Periodicals] Starting [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] periodical in [60s], polling every [180s].
2021-09-29T14:15:34.481Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2021-09-29T14:15:37.579Z INFO  [NetworkListener] Started listener bound to [10.1.3.163:9000]
2021-09-29T14:15:37.580Z INFO  [HttpServer] [HttpServer] Started.
2021-09-29T14:15:37.580Z INFO  [JerseyService] Started REST API at <10.1.3.163:9000>
2021-09-29T14:15:37.582Z INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, UserSessionTerminationService [RUNNING]=21, GracefulShutdownService [RUNNING]=69, LocalKafkaMessageQueueWriter [RUNNING]=69, UrlWhitelistService [RUNNING]=69, LocalKafkaMessageQueueReader [RUNNING]=70, LocalKafkaJournal [RUNNING]=74, OutputSetupService [RUNNING]=77, JobSchedulerService [RUNNING]=77, PrometheusExporter [RUNNING]=79, BufferSynchronizerService [RUNNING]=80, MongoDBProcessingStatusRecorderService [RUNNING]=100, EtagService [RUNNING]=122, ConfigurationEtagService [RUNNING]=131, StreamCacheService [RUNNING]=205, LookupTableService [RUNNING]=215, PeriodicalsService [RUNNING]=245, JerseyService [RUNNING]=3347}
2021-09-29T14:15:37.583Z INFO  [ServiceManagerListener] Services are healthy
2021-09-29T14:15:37.583Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from UninitializedÂ [LB:DEAD] to RunningÂ [LB:ALIVE]
2021-09-29T14:15:37.588Z INFO  [ServerBootstrap] Graylog server up and running.
2021-09-29T14:15:37.588Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-09-29T14:15:37.610Z INFO  [InputStateListener] Input [Syslog UDP/6118f4544e7ad752469d44ec] is now STARTING
2021-09-29T14:15:37.611Z INFO  [InputStateListener] Input [GELF UDP/6115f9304e7ad7524699f1e0] is now STARTING
2021-09-29T14:15:38.194Z WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0x84244b1d]'
2021-09-29T14:15:38.194Z WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0xb8717ac6]'
2021-09-29T14:15:38.231Z WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Linux Syslog UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=086ea892-b443-4092-aff8-ca9e3cdf0f36} (channel [id: 0xb8717ac6, L:/0:0:0:0:0:0:0:0:8514]) should be 262144 but is 212992.
2021-09-29T14:15:38.230Z WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=Windows UDP, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=086ea892-b443-4092-aff8-ca9e3cdf0f36} (channel [id: 0x84244b1d, L:/10.1.3.163:12201]) should be 262144 but is 212992.
2021-09-29T14:15:38.247Z INFO  [InputStateListener] Input [GELF UDP/6115f9304e7ad7524699f1e0] is now RUNNING
2021-09-29T14:15:38.250Z INFO  [InputStateListener] Input [Syslog UDP/6118f4544e7ad752469d44ec] is now RUNNING

But if I curl localhost:9000, I get “connection refused”:

curl: (7) Failed to connect to localhost port 9000: Connection refused

I can curl localhost:9200 and elasticsearch responds, so this looks like an issue with graylog. Let me know what else I can check.

hcomtes · September 29, 2021, 2:24pm

I changed the port to 9000, and the application appears to start, but if I curl localhost:9000 I get “connection refused”. I can curl localhost:9200 I get elasticsearch responding, so I think this is an issue with graylog.

The log says the graylog server is up and running, and the service is active and running.

I tried responding with the log, but Akismet blocked the post.

aaronsachs · September 29, 2021, 3:40pm

Hey there. Catching up on the post, I’ll note that using :80 as your port is an antipattern. Anything below port 1024 is a privileged port and can’t be used without running Graylog as root, which we strongly recommend against unless you have a VERY good reason to do so.

When it comes to curling localhost:9000, you should see something like:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="robots" content="noindex, nofollow">
    <meta charset="UTF-8">
    <title>Graylog Web Interface</title>
    <link rel="shortcut icon" href="/assets/favicon.png">

  </head>
  <body>
    <script src="/config.js"></script>

    <script src="/assets/vendor.53480e037b0a599cf54a.js"></script>

    <script src="/assets/polyfill.909253c37b9ca88e70e0.js"></script>

    <script src="/assets/builtins.909253c37b9ca88e70e0.js"></script>

    <script src="/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.a0abb3099d310b99f981.js"></script>

    <script src="/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.b63e75ab4cd9f320dbfe.js"></script>

    <script src="/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.0fdc66dc14c3d97376e9.js"></script>

    <script src="/assets/plugin/org.graylog.integrations.IntegrationsPlugin/plugin.org.graylog.integrations.IntegrationsPlugin.ddab5280b42173b8f8fe.js"></script>

    <script src="/assets/plugin/org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin/plugin.org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin.f2c44f3a982ece74b1b2.js"></script>

    <script src="/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.ba0c1ef29d3af8acf6f7.js"></script>

    <script src="/assets/app.909253c37b9ca88e70e0.js"></script>

  </body>
</html>

The fact that you don’t tells me that there’s something else at play here, maybe a firewall rule? Have you tried running netstat -ntlp | grep 9000 to see if Graylog is actually up and listening? I

f it’s showing up, then you should try running nc -vz localhost 9000 to see if you can successfully make a connection to Graylog. If that works, then I’d see if you have ufw running by using ufw status.

If none of those work, you’ll need to share your config.

hcomtes · September 29, 2021, 9:19pm

I wasn’t aware running on port 80 wasn’t recommended, that’s fine, I can use a different port. My main issue is I’m still not getting a response through curl.

When I run sudo netstat -ntlp | grep 9000 I get:
tcp6 0 0 10.1.3.163:9000 :::* LISTEN 1380917/java

And for nc -vz localhost 9000 I get:
nc: connect to localhost port 9000 (tcp) failed: Connection refused

As I mentioned before I can curl 9200 and get a response from elastic search, so I think this is a graylog issue.

tmacgbay · September 29, 2021, 9:43pm

I get connection refused if I use localhost but if I use the hostname I get:

$ nc -vz G-Serv 9000
Connection to G-Serv 9000 port [tcp/*] succeeded!

same issue with curl. I have to use the hostname and not localhost.

I don’t see anywhere in the server.conf file you posted earlier where the elasticsearch_hosts is defined…

# List of Elasticsearch hosts Graylog should connect to.
# Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
# If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
# requires authentication.
#
# Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

If it’s on the same machine you may still need to define it and use the server name. Just guessing.

Watch Graylog logs when you start the service

tail -f /var/log/graylog-server/server.log

post anything new since you now have the service running…

aaronsachs · September 29, 2021, 10:16pm

It just occurred to me that I think your mongodb_uri is incorrect. You currently have it set to:

mongodb_uri = mongodb://localhost/graylog

But given that MongoDB runs on 27017, your port is missing. It should look like:

mongodb_uri = mongodb://localhost:27017/graylog

This is likely why the curl’s failing–the API is attempting to start, but given that it’s not successfully making a connection to MongoDB, it’s not actually responding.

hcomtes · September 29, 2021, 11:20pm

I get connection refused if I use localhost
That’s really odd, but it seems to be the issue. If I use the IP with curl it works.

So everything seems to be working fine on port 9000 now, but I’d like to have the site on port 80. I know you don’t recommend the application running on port 80. So what’s the best practice for achieving this from your perspective?

aaronsachs · September 30, 2021, 2:02am

Best practice is to use a reverse proxy like Apache or Nginx. I’ve posted some Nginx examples here Nginx Config Examples

system · October 14, 2021, 2:03am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog startup failed Graylog Central (peer support)	4	3320	January 12, 2018
Graylog startup failed - JerseyService FAILED Graylog Central (peer support)	4	912	May 2, 2023
JerseyService Failing Graylog Central (peer support)	4	1251	September 6, 2022
Graylog failed to start after yum updates Graylog Central (peer support)	3	376	December 18, 2018
Graylog crashes on startup if HTTPS is enabled Graylog Central (peer support)	20	1147	May 8, 2020

Graylog startup failed - Java IllegalStateException

Description of your problem

Description of steps you’ve taken to attempt to solve the issue

Environmental information

Package versions

Related topics