Graylog Server Started Successfully, But Web Page Not Opening

Graylog Central (peer support)
1

Hello,

I installed Graylog on Ubuntu 22.04 server about a month ago, and follwoing the recommendation on the official installation page, I installed OpenSearch instead of Elasticsearch, and it’s been working great until last night when I ran apt-get update (I’m not entirely sure it’s the reason for this problem).

Here are the information/troubleshooting I have so far.

dpkg -l | grep -E ".(opensearch|graylog|mongo)."
ii  graylog-5.1-repository                1-2                                     all          Package to install Graylog 5.1 GPG key and repository
ii  graylog-server                        5.1.2-1                                 amd64        Graylog server
ii  mongodb-database-tools                100.7.2                                 amd64        mongodb-database-tools package provides tools for working with the MongoDB server:
ii  mongodb-mongosh                       1.10.0                                  amd64        MongoDB Shell CLI REPL Package
ii  mongodb-org                           6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database                  6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database-tools-extra      6.0.6                                   amd64        Extra MongoDB database tools
ii  mongodb-org-mongos                    6.0.6                                   amd64        MongoDB sharded cluster query router
ii  mongodb-org-server                    6.0.6                                   amd64        MongoDB database server
ii  mongodb-org-shell                     6.0.6                                   amd64        MongoDB shell client
ii  mongodb-org-tools                     6.0.6                                   amd64        MongoDB tools
ii  opensearch                            2.8.0                                   amd64        An open source distributed and RESTful search engine


service graylog-server status
● graylog-server.service - Graylog server
     Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-06-16 11:26:47 +03; 30min ago
       Docs: http://docs.graylog.org/
   Main PID: 3902 (graylog-server)
      Tasks: 44 (limit: 43216)
     Memory: 226.4M
        CPU: 14.174s
     CGroup: /system.slice/graylog-server.service
             ├─3902 /bin/sh /usr/share/graylog-server/bin/graylog-server
             └─3903 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseN>

Jun 16 11:26:47 dell-poweredge-r420 systemd[1]: Started Graylog server.


netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      1018/mongod
tcp        0      0 127.0.0.1:8088          0.0.0.0:*               LISTEN      1055/influxd
tcp        0      0 127.0.0.1:33060         0.0.0.0:*               LISTEN      1319/mysqld
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      985/systemd-resolve
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1319/mysqld
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1117/sshd: /usr/sbi
tcp        0      0 0.0.0.0:10051           0.0.0.0:*               LISTEN      1744/zabbix_server
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1103/zabbix_agentd
tcp6       0      0 ::1:9300                :::*                    LISTEN      3014/java
tcp6       0      0 ::1:9200                :::*                    LISTEN      3014/java
tcp6       0      0 :::8086                 :::*                    LISTEN      1055/influxd
tcp6       0      0 :::3000                 :::*                    LISTEN      1709/grafana
tcp6       0      0 :::80                   :::*                    LISTEN      1239/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      1117/sshd: /usr/sbi
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      3014/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      3014/java
tcp6       0      0 :::10051                :::*                    LISTEN      1744/zabbix_server
tcp6       0      0 :::10050                :::*                    LISTEN      1103/zabbix_agentd


 tail /var/log/graylog-server/server.log
2023-06-16T11:58:37.856+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:37.857+03:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #382
2023-06-16T11:58:42.861+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:42.862+03:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #383
2023-06-16T11:58:47.866+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:47.867+03:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #384
2023-06-16T11:58:52.871+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:52.872+03:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #385
2023-06-16T11:58:57.876+03:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/... - \n not found: limit=0 content=….
2023-06-16T11:58:57.877+03:00 INFO  [VersionProbe] Elasticsearch is not available. Retry #386

server.conf:

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = <sha-hash>
root_username = admin
root_password_sha2 = <sha-hash>
root_email = "<myemail@example.com>"
root_timezone = Cont/City
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.1:9000
stream_aware_field_types=false
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
transport_email_enabled = true
transport_email_hostname = smtp.office365.com
transport_email_port = 587
transport_email_use_auth = true
transport_email_auth_username = <myemail@example.com>
transport_email_auth_password = <mypassword>
transport_email_from_email = <myemail@example.com>
transport_email_use_tls = true

It might be worth mentioning that after the problem started I ran apt-get upgrade. While doing so I got this message:

Configuration file '/etc/opensearch/opensearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ?

My answer was Y.

The server was rebooted after the upgrade was done.

Please let me know if more info is required.

2

I think here is your problem. Elastic is not running as it should. Did you try to migrate to Opensearch, and also start the service?

3

I never used Elasticsearch. I installed Graylog with OpenSearch as recommended on the official installation page.

4

That’s good news, so you will need to make you Graylog talk to Opensearch again. It looks as they to not talk to each other.
Can you check if the service is running in the first place? Your netstat-command looks like, but to be sure.
Next check your Graylog-Config, if the right IPs/Ports and users are in there.

5

Here’s the output of Opensearch status:

service opensearch status
● opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-06-16 11:14:36 +03; 1h 27min ago
       Docs: https://opensearch.org/
   Main PID: 3014 (java)
      Tasks: 182 (limit: 43216)
     Memory: 8.9G
        CPU: 5min 36.734s
     CGroup: /system.slice/opensearch.service
             └─3014 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.>

Jun 16 11:14:20 dell-poweredge-r420 systemd[1]: Starting OpenSearch...
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch>
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Ope>
Jun 16 11:14:23 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager will be removed in a future release
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (>
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Sec>
Jun 16 11:14:24 dell-poweredge-r420 systemd-entrypoint[3014]: WARNING: System::setSecurityManager will be removed in a future release
Jun 16 11:14:36 dell-poweredge-r420 systemd[1]: Started OpenSearch.

The server IP address and users are correct, it’s been working fine for over a month and I haven’t changed anything in server.conf.

But even if there’s a problem in the communication between Graylog and Opensearch, isn’t the webpage supposed to at least open and ask for credentials?

6

I solved the problem. Apparently this was the culprit:

I found opensearch.yml reset to the default template and all my config lines were gone. I added the original config as shown below, after which the server ran without any problem.

cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
discovery.type: single-node
network.host: 0.0.0.0
action.auto_create_index: false
plugins.security.disabled: true
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.