Problem retrieving data for widget

fvr_flho · June 2, 2023, 11:50am

**

:**

I’m not sure if there’s a connection, but shortly after I have changed the time on my DC’s from 12-hour to 24-hour time format, the Graylog server started to display the message shown in attached picture everywhere within the searh and dashboards fields (we do have a filebeat getting DNS debug logs from the DC’s)
The node is still active and is processing messages without any problems.

2. Describe your environment:

OS Information: OpenSUSE Leap 15.4
Package Version: 5.1.1+ef1b993
Not sure what logs whould be relevant - as I can see no errors (in fact no entries) in graylog nor opensearch logs when I go into the different dashboards (or making a search)

3. What steps have you already taken to try and solve the problem?

I’m lost - so I asked my friend Google… no positive result

4. How can the community help?

Everything would be helpful

Best regards

/Flemming

gsmith · June 2, 2023, 9:25pm

Hey @fvr_flho

Couple questions,

Have you tried a different browser? If so do you get same results?
Hows the status of Graylog. MongoDb and Opensearch ( I assume you using that)?
Have you tried to create a new widget or is this the default? if so do you get the same results?
How did you configure Opensearch && Graylog? Are you bale to show the configuration files for each?
What documentation did you use to install Graylog/OS/MongoDb?

fvr_flho · June 5, 2023, 7:30am

Hi @gsmith

Have you tried a different browser? If so do you get same results?

I have tried both Edge (Chromium) & Firefox, both with same results
I have tried with the built in root user, and an LDAP enabled user, both with same results.

Hows the status of Graylog. MongoDb and Opensearch ( I assume you using that)?

Correctly assumed - status as follows:

mygraylog:~ # systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:35 CEST; 2 days ago
Docs: http://docs.graylog.org/
Main PID: 1942 (graylog-server)
Tasks: 317 (limit: 4915)
CGroup: /system.slice/graylog-server.service
├─ 1942 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─ 1945 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -jar -Dlog4j.configurationFile=f>
Notice: journal has been rotated since unit was started, output may be incomplete.

mygraylog:~ # systemctl status mongod.service
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:35 CEST; 2 days ago
Docs: https://docs.mongodb.org/manual
Main PID: 1943 (mongod)
CGroup: /system.slice/mongod.service
└─ 1943 /usr/bin/mongod -f /etc/mongod.conf
Notice: journal has been rotated since unit was started, output may be incomplete.

mygraylog:~ # systemctl status opensearch.service
● opensearch.service - OpenSearch
Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:37 CEST; 2 days ago
Docs: https://opensearch.org/
Main PID: 1089 (java)
Tasks: 112 (limit: 4915)
CGroup: /system.slice/opensearch.service
└─ 1089 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=U>
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:179)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:217)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.jav>
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.lang.Thread.run(Thread.java:833)
lines 1-19/19 (END)

Have you tried to create a new widget or is this the default? if so do you get the same results?

Same result if I use the default widget, an imported from Marketplace and if I create a new one.

How did you configure Opensearch && Graylog? Are you bale to show the configuration files for each?

Below I have provided the uncommented sections of the config files:

/etc/opensearch/opensearch.yml:
cluster.name: mygraylog.dom.name
path.data: /data/opensearch
path.logs: /var/log/opensearch

/etc/graylog/server/server.conf:
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret =
root_username =
root_password_sha2 =
bin_dir = /usr/share/graylog-server/bin
data_dir = /data/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.10.21:9000
http_enable_tls = true
http_tls_cert_file = //mygraylog.dom.name.fullchained.crt
http_tls_key_file = //mygraylog.dom.name.pkcs8.key
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /data/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 25gb
message_journal_flush_age = 5m
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000

What documentation did you use to install Graylog/OS/MongoDb?

https://go2docs.graylog.org/5-0/downloading_and_installing_graylog/suse_installation.htm

Best regards
/Flemming

fvr_flho · June 6, 2023, 7:32am

Hi

By the way, I have installed the following from marketplace:

https://community.graylog.org/t/windows-dns-for-graylog/22999

And according to the documentation for this, I have made the following change:

curl -XPUT localhost:9200/_index_template/graylog -H ‘Content-Type: application/json’ -d ’
{
“index_patterns”: [“graylog*”],
“template”: {
“settings”: {
“index.refresh_interval”: “30s”
},
“mappings”: {
“properties”: {
“ThreadID”: {
“type”: “keyword”
}
}
}
}
}’

Best regards
/Flemming

fvr_flho · June 6, 2023, 9:53am

Hi

Just as information - tried to remove the setting made previously with:

curl -XDELETE localhost:9200/_index_template/graylog -H ‘Content-Type: application/json’

Didn’t help - still no possibility to use Graylog at all

gsmith · June 7, 2023, 2:06am

Hey @fvr_flho

Thanks for the info. First thing I would suggest is not use the plugin and see if you can create your own widget.
I spotted a couple things.Your graylog config file does not seem correct.

Graylog configuration file.

Your configuration.

http_bind_address = 192.168.10.21:9000
http_enable_tls = true
http_tls_cert_file = //mygraylog.dom.name.fullchained.crt
http_tls_key_file = //mygraylog.dom.name.pkcs8.key

My configuration

[root@graylog graylog_user]# cat /etc/graylog/server/server.conf  | egrep -v "^\s*(#|$)"
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = epOqmLi7r7CdZxl76QOQxr8bRUP
root_password_sha2 = 5e884898da28047151d0e56f8dc6
root_email = "greg.smith@domain.com"
root_timezone = America/Chicago
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.100:9000
http_publish_uri = https://graylog.doamin.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Opensearch Configuration

If you using localhost in connecting Graylog to Opensearch. Here is basic configurations

cluster.name: graylog
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: localhost {and/or 127.0.0.1}
http.port: 9200
action.auto_create_index: false
discovery.type: single-node
bootstrap.memory_lock: true
plugins.security.disabled: true
plugins.security.system_indices.enabled: false

If you can run Graylog without certificates (i.e., HTTP) then either its your cretificates or configurations made. Dont forget to check firewall or Apparmor/Selinux , just in case.

If you followed this documentation for certificates.

Using HTTPS

Then you should have these two certificates…

hope that helps

fvr_flho · June 7, 2023, 8:08am

Hi @gsmith

I think the only difference between us, is the naming of the files… my .crt & .key is the same as .pem in your setup (old habit from my side, I prefer the naming on order to differ between the key and the cert)

I have tried without SSL, and the problem remains - So I suspect the upgrade has messed something up in OepnSearch… maybe I should open a bug

Best regards
/Flemming

system · June 21, 2023, 8:09am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog Dashboard Loading Problem (waits without any error) Graylog Central (peer support)	16	1774	August 28, 2020
GrayLog Crashing Graylog Central (peer support)	9	2102	November 22, 2018
Graylog Server Started Successfully, But Web Page Not Opening Graylog Central (peer support)	6	959	June 30, 2023
Graylog unable to access on web Graylog Central (peer support)	4	1109	May 30, 2019
Graylog logs indicate no errors and server running; web site not responsive Graylog Central (peer support)	11	7753	February 12, 2018

Problem retrieving data for widget

Related Topics