Graylog 5.0.6 fails to bind to port 9000

flyshoo · April 22, 2023, 3:42am

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Graylog 5 starts and all of it’s dependancies but doesn’t bind to tcp port 9000.
tcp6 0 0 ip6-localhost:9200 [::]:* LISTEN 1177/java
tcp6 0 0 localhost:9200 [::]:* LISTEN 1177/java
tcp6 0 0 localhost:9300 [::]:* LISTEN 1177/java
tcp6 0 0 ip6-localhost:9300 [::]:* LISTEN 1177/java

It only happens after a recent update to Ubuntu 22.04.
I can go back to my last TimeShift snapshot and Graylog starts and binds to tcp port 9000.

2. Describe your environment:

OS Information:
Ubuntu 22.04
Package Version:
5.0.6
Service logs, configurations, and environment variables:
Graylog.log messages
root@ces-linux-01:~# grep ERROR /var/log/elasticsearch/graylog.log

[2023-04-21T08:22:18,005][ERROR][o.e.i.g.GeoIpDownloader ] [ces-linux-01] exception during geoip databases update

[2023-04-21T08:58:47,764][ERROR][o.e.i.g.GeoIpDownloader ] [ces-linux-01] exception during geoip databases update

root@ces-linux-01:~# grep WARNING /var/log/elasticsearch/graylog.log

root@ces-linux-01:~# grep WARN /var/log/elasticsearch/graylog.log

[2023-04-21T08:20:33,720][WARN ][o.e.c.InternalClusterInfoService] [ces-linux-01] failed to retrieve stats for node [L8TAII3STuKssvoatbKtQA]: [ces-linux-01][127.0.0.1:9300][cluster:monitor/nodes/stats[n]]

[2023-04-21T08:20:33,744][WARN ][o.e.c.InternalClusterInfoService] [ces-linux-01] failed to retrieve shard stats from node [L8TAII3STuKssvoatbKtQA]: [ces-linux-01][127.0.0.1:9300][indices:monitor/stats[n]]

[2023-04-21T08:22:13,843][WARN ][o.e.b.BootstrapChecks ] [ces-linux-01] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

[2023-04-21T08:22:16,487][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [ces-linux-01] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See Set up minimal security for Elasticsearch | Elasticsearch Guide [7.17] | Elastic to enable security.

[2023-04-21T08:58:46,029][WARN ][o.e.b.BootstrapChecks ] [ces-linux-01] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

[2023-04-21T08:58:47,203][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [ces-linux-01] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See Set up minimal security for Elasticsearch | Elasticsearch Guide [7.17] | Elastic to enable security.

Graylog server.log
2023-04-21T08:20:30.802-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #1).

2023-04-21T08:20:30.809-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #2).

2023-04-21T08:20:30.814-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #3).

2023-04-21T08:20:30.824-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #4).

2023-04-21T08:20:30.842-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #5).

2023-04-21T08:20:30.876-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #6).

2023-04-21T08:20:30.942-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #7).

2023-04-21T08:20:31.072-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #8).

2023-04-21T08:20:31.330-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #9).

2023-04-21T08:20:31.844-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #10).

2023-04-21T08:20:32.870-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #11).

2023-04-21T08:20:34.920-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #12).

2023-04-21T08:20:35.890-07:00 ERROR [ClusterAdapterES7] An error occurred: (Connection refused)

2023-04-21T08:20:35.984-07:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.

2023-04-21T08:20:39.019-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #13).

2023-04-21T08:20:40.985-07:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.

2023-04-21T08:20:45.986-07:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.

2023-04-21T08:20:47.213-07:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: ConnectException[Connection refused]; nested: ConnectException[Connection refused];, errorDetails=}, retrying (attempt #14).

2023-04-21T08:20:50.987-07:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.

2023-04-21T08:20:55.989-07:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused.

2023-04-21T08:20:57.104-07:00 ERROR [InMemoryRolePermissionResolver] Could not find roles collection, no user roles updated.

2023-04-21T08:20:57.804-07:00 ERROR [JobSchedulerService] Error running job execution engine

2023-04-21T08:20:57.914-07:00 ERROR [NodePingThread] Uncaught exception in Periodical

2023-04-21T08:20:57.983-07:00 ERROR [MongoDBProcessingStatusRecorderService] Couldn’t persist processing status

3. What steps have you already taken to try and solve the problem?

Google searches for the log entry but nothing that works.

4. How can the community help?

Some assist how to debug the issue or suggestion on what could be the issue.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

flyshoo · April 22, 2023, 3:50am

Here is my graylog server configuration.

root@ces-linux-01:~# cat /etc/graylog/server/server.conf | egrep -v “^\s*(#|$)”

is_leader = true

node_id_file = /etc/graylog/server/node-id

password_secret = REDACTED

root_password_sha2 = REDACTED

root_timezone = PST8PDT

bin_dir = /usr/share/graylog-server/bin

data_dir = /var/lib/graylog-server

plugin_dir = /usr/share/graylog-server/plugin

stream_aware_field_types=false

rotation_strategy = count

elasticsearch_max_docs_per_index = 20000000

elasticsearch_max_number_of_indices = 20

retention_strategy = delete

elasticsearch_shards = 4

elasticsearch_replicas = 0

elasticsearch_index_prefix = graylog

allow_leading_wildcard_searches = false

allow_highlighting = false

elasticsearch_analyzer = standard

output_batch_size = 500

output_flush_interval = 1

output_fault_count_threshold = 5

output_fault_penalty_seconds = 30

processbuffer_processors = 5

outputbuffer_processors = 3

processor_wait_strategy = blocking

ring_size = 65536

inputbuffer_ring_size = 65536

inputbuffer_processors = 2

inputbuffer_wait_strategy = blocking

message_journal_enabled = true

message_journal_dir = /var/lib/graylog-server/journal

lb_recognition_period_seconds = 3

mongodb_uri = mongodb://localhost/graylog

mongodb_max_connections = 1000

integrations_scripts_dir = /usr/share/graylog-server/scripts

flyshoo · April 22, 2023, 4:35am

It binds to the ipv6 address @ces-linu tcp6 0 0 localhost:9200 tcp6 0 0 localhost:9000 tcp6 0 0 localhost:9300 tcp6 0 0 [::]:shell tcp6 0 0 ip6-localhost:9200 tcp6 0 0 ip6-localhost:9300 tcp6 0 0 [::]:12201 udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:syslog udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 udp6 0 0 [::]:1514 even when it comes online. Don’t know if that has anything to do with it.
x-01:~$ sudo netstat -tulp | grep java
[::]:* LISTEN 1171/java
[::]:* LISTEN 1175/java
[::]:* LISTEN 1171/java
[::]:* LISTEN 1175/java
[::]:* LISTEN 1171/java
[::]:* LISTEN 1171/java
[::]:* LISTEN 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java
[::]:* 1175/java

flyshoo · April 22, 2023, 4:42am

One more detail, it is behind an nginx proxy. That log just has an error no upstream neighbor or something.

gsmith · April 22, 2023, 5:22am

Hey @flyshoo
I dont see this configured in your file

http_bind_address = 127.0.0.1:9000

Or you can use this since your behind nginx proxy.

http_bind_address = 0.0.0.0:9000

flyshoo · April 22, 2023, 6:28pm

@gsmith thanks for the suggestion.

I added that line and disabled elasticsearch version check because it was complaining failing to determine version. Even set the major version of elasticsearch.
Still fails to bind to tcp port 9000 on any address.

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = REDACTED
root_password_sha2 = REDACTED
root_timezone = PST8PDT
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_disable_version_check = true
elasticsearch_version = 7
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts

flyshoo · April 23, 2023, 4:05am

I rolled back the updates and everything is working. I’m going to do a binary search for the offending update and go from there.

Thanks, for your views and I’ll open another ticket after I find the update.
Flyshoo

flyshoo · April 24, 2023, 2:53am

Appears the new version of graylog is the issue.

Is it possible to start graylog in debug mode?

flyshoo · April 24, 2023, 3:18am

Do these messages from systemctl mean anything?
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:568)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at com.google.inject.Guice.createInjector(Guice.java:87)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:502)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:260)
Apr 23 20:01:04 ces-linux-01 graylog-server[2706]: at org.graylog2.bootstrap.Main.main(Main.java:45)

flyshoo · April 24, 2023, 3:49am

Ok, I resolved the issue. The update wrote over the systemd file. I’m running the service as root. I know that is not recommended but it works.

gsmith · April 24, 2023, 9:10pm

Hey @flyshoo

Sorry for the late reply. glad to see you resolved it

system · May 8, 2023, 9:11pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog Web - Port 9000 - Not Working Graylog Central (peer support)	2	1976	June 21, 2022
Graylog 5.2 not binding to port 9000 on Ubuntu 20.04 Graylog Central (peer support)	2	48	April 27, 2024
Graylog 5.1.2 white blank web interface Graylog Central (peer support) dashboards	3	467	July 27, 2023
Upgrade from 5.0.9 to 5.1.5 No Web UI Graylog Central (peer support)	8	377	November 1, 2023
Cannot upgrade past 5.0.9 Graylog Central (peer support)	29	478	December 26, 2023

Graylog 5.0.6 fails to bind to port 9000

Related Topics