New Ubuntu install - initial setup prompting to configure opensearch?

brannen · May 16, 2024, 3:14pm

I just completed following the instructions to install Graylog 5.2 on a new Ubuntu server with Opensearch and Mongodb.

Upon connecting to the webpage, it opens “Graylog Initial Setup” and wants me to implement a “Data Node” - with an error, “No data nodes have been found.”

Graylog /var/log/graylog-server/server.log reports this error, repeating.

2024-05-16T14:01:01.117Z ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://127.0.0.1:9200/… - \n not found: limit=0 content=….
2024-05-16T14:01:01.118Z INFO [VersionProbe] OpenSearch/Elasticsearch is not available. Retry #1

I’ve tried stopping and restarting services to no avail.

2. Describe your environment:

OS Information:

Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

Package Version:
dpkg -l | grep -E “.(elasticsearch|graylog|mongo).”
ii graylog-5.2-repository 1-2 all Package to install Graylog 5.2 GPG key and repository
ii graylog-enterprise 5.2.7-1 amd64 Graylog Enterprise Server
ii mongodb-database-tools 100.9.4 amd64 mongodb-database-tools package provides tools for working with the MongoDB server:
hi mongodb-mongosh 2.2.5 amd64 MongoDB Shell CLI REPL Package
hi mongodb-org 7.0.9 amd64 MongoDB open source document-oriented database system (metapackage)
hi mongodb-org-database 7.0.9 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-database-tools-extra 7.0.9 amd64 Extra MongoDB database tools
hi mongodb-org-mongos 7.0.9 amd64 MongoDB sharded cluster query router
hi mongodb-org-server 7.0.9 amd64 MongoDB database server
ii mongodb-org-shell 7.0.9 amd64 MongoDB shell client
hi mongodb-org-tools 7.0.9 amd64 MongoDB tools
Service logs, configurations, and environment variables:
relevant portions of config file.
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret =12D3XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXv0BY-
root_username = ADMIN
root_password_sha2 =9cXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX692 -
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.1.0.13:9000
http_publish_uri = http://10.1.0.13:9000/
stream_aware_field_types=false
disabled_retention_strategies = none
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts

3. What steps have you already taken to try and solve the problem?
Google, stop/restart services, check config file.

4. How can the community help?
advise how to solve the problem?

gsmith · May 16, 2024, 10:42pm

Hey @brannen

try commenting out the line

http_publish_uri = http://10.1.0.13:9000/

By chance did you leave OpenSearch Config file default?

network.host: 127.0.0.1

brannen · May 17, 2024, 2:56pm

Thanks @gsmith - I commented out the publish_uri like you said. I don’t know where the Opensearch Config file is, so I assume it’s default.

I restated the server with systemctl - still getting the same webpage, with these additional error messages.

There was an error fetching the data nodes: There was an error fetching a resource: undefined. Additional information: Not available

There was an error fetching the certificate authority: There was an error fetching a resource: undefined. Additional information: Not available

There was an error fetching the renewal policy: There was an error fetching a resource: undefined. Additional information: Not available

brannen · May 17, 2024, 2:59pm

Clicked on “resume startup”, which said it would cause the config to fail back to using Opensearch instance on localhost:9200.

Got error:

Could not resume startup

Resuming startup failed with error: FetchError: There was an error fetching a resource: undefined. Additional information: Not available

gsmith · May 17, 2024, 9:52pm

Hey @brannen

Where are you see this error at?

Addition steps.
Check the status of " OpenSearch, Graylog and MongoDb"
Check OpenSearch logs for any errors or warnings.

If possible can you show you OpenSearch Config file?

brannen · May 17, 2024, 10:23pm

I looked for the OpenSearch config file. I don’t know where it is.

brannen · May 17, 2024, 10:33pm

when I go to the http page.

brannen · May 17, 2024, 10:34pm

then click on “resume startup” and get message:

brannen · May 17, 2024, 10:34pm

then get a pop up in bottom right of browser.

gsmith · May 17, 2024, 10:53pm

Its the connection between Graylog and OpenSearch.

What does you OpenSearch configuration look like?

brannen · May 17, 2024, 11:10pm

Where can I find the opensearch configuration file?

gsmith · May 18, 2024, 12:07am

From this step here

brannen · May 19, 2024, 3:13pm

Here’s the uncommented config.

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3

brannen · May 19, 2024, 3:33pm

I changed the file to this - only the uncommented lines.

cluster.name: graylog
node.name: $(HOSTNAME)
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
discovery.type: single-node
action.auto_crete_index: false
plugins.security.disabled: true
indicies.query.bool.max_clause_count: 32768

brannen · May 19, 2024, 3:42pm

Now getting this, when I try to start the service


admin@aznc-graylog-02:~$ sudo systemctl start opensearch.service



Job for opensearch.service failed because a timeout was exceeded.
See "systemctl status opensearch.service" and "journalctl -xe" for details.

admin@aznc-graylog-02:~$ sudo systemctl status opensearch.service
● opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Sun 2024-05-19 15:37:43 UTC; 1min 14s ago
       Docs: https://opensearch.org/
    Process: 33051 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status>
   Main PID: 33051 (code=exited, status=143)

May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:1>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(Environmen>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.j>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.Command.main(Command.java:101)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]: For complete error details, refer to the log at /var/log/opensearch/gray>
May 19 15:37:43 aznc-graylog-02 systemd[1]: opensearch.service: start operation timed out. Terminating.
May 19 15:37:43 aznc-graylog-02 systemd[1]: opensearch.service: Failed with result 'timeout'.
May 19 15:37:43 aznc-graylog-02 systemd[1]: Failed to start OpenSearch.

admin@aznc-graylog-02:~$ sudo journalctl -xe
-- 
-- The unit opensearch.service has successfully entered the 'dead' state.
May 19 15:36:23 aznc-graylog-02 systemd[1]: Stopped OpenSearch.
-- Subject: A stop job for unit opensearch.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A stop job for unit opensearch.service has finished.
-- 
-- The job identifier is 21369 and the job result is done.
May 19 15:36:23 aznc-graylog-02 sudo[33027]: pam_unix(sudo:session): session closed for user root
May 19 15:36:28 aznc-graylog-02 sudo[33048]: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/systemctl sta>
May 19 15:36:28 aznc-graylog-02 sudo[33048]: pam_unix(sudo:session): session opened for user root by admin(uid=0)
May 19 15:36:28 aznc-graylog-02 systemd[1]: Starting OpenSearch...
-- Subject: A start job for unit opensearch.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit opensearch.service has begun execution.
-- 
-- The job identifier is 21370.
May 19 15:36:29 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: Using incubator modules: jdk.incubator.vector
May 19 15:36:29 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: A terminally deprecated method in java.lang.System has been cal>
May 19 15:36:29 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: System::setSecurityManager has been called by org.opensearch.bo>
May 19 15:36:29 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: Please consider reporting this to the maintainers of org.opense>
May 19 15:36:29 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: System::setSecurityManager will be removed in a future release
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: May 19, 2024 3:36:30 PM sun.util.locale.provider.LocaleProviderAdapter <>
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: COMPAT locale provider will be removed in a future release
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: A terminally deprecated method in java.lang.System has been cal>
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: System::setSecurityManager has been called by org.opensearch.bo>
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: Please consider reporting this to the maintainers of org.opense>
May 19 15:36:30 aznc-graylog-02 systemd-entrypoint[33051]: WARNING: System::setSecurityManager will be removed in a future release
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]: uncaught exception in thread [main]
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]: java.lang.IllegalArgumentException: unknown setting [indicies.query.bool>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.common.settings.AbstractScopedSettings.validat>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.common.settings.AbstractScopedSettings.validat>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.common.settings.AbstractScopedSettings.validat>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.common.settings.AbstractScopedSettings.validat>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.common.settings.SettingsModule.<init>(Settings>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.node.Node.<init>(Node.java:594)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.node.Node.<init>(Node.java:420)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:24>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:1>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(Environmen>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.j>
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.cli.Command.main(Command.java:101)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
May 19 15:36:33 aznc-graylog-02 systemd-entrypoint[33051]: For complete error details, refer to the log at /var/log/opensearch/gray>
May 19 15:37:43 aznc-graylog-02 systemd[1]: opensearch.service: start operation timed out. Terminating.
May 19 15:37:43 aznc-graylog-02 systemd[1]: opensearch.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit opensearch.service has entered the 'failed' state with result 'timeout'.
May 19 15:37:43 aznc-graylog-02 systemd[1]: Failed to start OpenSearch.
-- Subject: A start job for unit opensearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit opensearch.service has finished with a failure.
-- 
-- The job identifier is 21370 and the job result is failed.
May 19 15:37:43 aznc-graylog-02 sudo[33048]: pam_unix(sudo:session): session closed for user root
May 19 15:38:58 aznc-graylog-02 sudo[33268]: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/systemctl sta>
May 19 15:38:58 aznc-graylog-02 sudo[33268]: pam_unix(sudo:session): session opened for user root by admin(uid=0)
May 19 15:39:46 aznc-graylog-02 sudo[33268]: pam_unix(sudo:session): session closed for user root
May 19 15:39:57 aznc-graylog-02 sudo[33271]: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/systemctl jou>
May 19 15:39:57 aznc-graylog-02 sudo[33271]: pam_unix(sudo:session): session opened for user root by admin(uid=0)
May 19 15:39:57 aznc-graylog-02 sudo[33271]: pam_unix(sudo:session): session closed for user root
May 19 15:40:04 aznc-graylog-02 sudo[33273]: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/journalctl -xe
May 19 15:40:04 aznc-graylog-02 sudo[33273]: pam_unix(sudo:session): session opened for user root by admin(uid=0)

gsmith · May 19, 2024, 6:08pm

Hey @brannen

It tell your in the log file what happened. this section here

unknown setting [indicies.query.bool>

Try commenting out that line and restart your service , see if that works.

brannen · May 20, 2024, 11:20am

Holy Crap! I think it worked!

I’m at a “Welcome to Graylog” login screen! Thanks for your help!!! @gsmith

patrickmann · May 21, 2024, 5:52am

@brannen You had a typo there, that’s why it was unknown. It should be indices.query.bool.max_clause_count

brannen · May 21, 2024, 10:28am

Thanks @patrickmann . It’s up and working now.

gsmith · May 21, 2024, 11:35pm

Hey @brannen

That’s awesome

Topic		Replies	Views
No data nodes have been found (using opensearch) Graylog Central (peer support)	10	3640	November 20, 2023
Graylog 5.1 and Opensearch 2.9 integration error Graylog Central (peer support)	7	1218	September 20, 2023
Why is the installation guide so confusing? Graylog Central (peer support) documentation	2	349	May 13, 2024
What changes are needed for fresh graylog 5. installation to enable opensearch / disable elasticsearch Graylog Central (peer support)	2	1335	January 17, 2023
Trouble installing net new single node install Graylog Central (peer support)	3	566	November 18, 2023

New Ubuntu install - initial setup prompting to configure opensearch?

Related topics