Problem retrieving data for widget

fvr_flho · June 2, 2023, 11:50am

**

:**

I’m not sure if there’s a connection, but shortly after I have changed the time on my DC’s from 12-hour to 24-hour time format, the Graylog server started to display the message shown in attached picture everywhere within the searh and dashboards fields (we do have a filebeat getting DNS debug logs from the DC’s)
The node is still active and is processing messages without any problems.

2. Describe your environment:

OS Information: OpenSUSE Leap 15.4
Package Version: 5.1.1+ef1b993
Not sure what logs whould be relevant - as I can see no errors (in fact no entries) in graylog nor opensearch logs when I go into the different dashboards (or making a search)

3. What steps have you already taken to try and solve the problem?

I’m lost - so I asked my friend Google… no positive result

4. How can the community help?

Everything would be helpful

Best regards

/Flemming

gsmith · June 2, 2023, 9:25pm

Hey @fvr_flho

Couple questions,

Have you tried a different browser? If so do you get same results?
Hows the status of Graylog. MongoDb and Opensearch ( I assume you using that)?
Have you tried to create a new widget or is this the default? if so do you get the same results?
How did you configure Opensearch && Graylog? Are you bale to show the configuration files for each?
What documentation did you use to install Graylog/OS/MongoDb?

fvr_flho · June 5, 2023, 7:30am

Hi @gsmith

Have you tried a different browser? If so do you get same results?

I have tried both Edge (Chromium) & Firefox, both with same results
I have tried with the built in root user, and an LDAP enabled user, both with same results.

Hows the status of Graylog. MongoDb and Opensearch ( I assume you using that)?

Correctly assumed - status as follows:

mygraylog:~ # systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:35 CEST; 2 days ago
Docs: http://docs.graylog.org/
Main PID: 1942 (graylog-server)
Tasks: 317 (limit: 4915)
CGroup: /system.slice/graylog-server.service
├─ 1942 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─ 1945 /usr/share/graylog-server/jvm/bin/java -Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -jar -Dlog4j.configurationFile=f>
Notice: journal has been rotated since unit was started, output may be incomplete.

mygraylog:~ # systemctl status mongod.service
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:35 CEST; 2 days ago
Docs: https://docs.mongodb.org/manual
Main PID: 1943 (mongod)
CGroup: /system.slice/mongod.service
└─ 1943 /usr/bin/mongod -f /etc/mongod.conf
Notice: journal has been rotated since unit was started, output may be incomplete.

mygraylog:~ # systemctl status opensearch.service
● opensearch.service - OpenSearch
Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-06-02 12:53:37 CEST; 2 days ago
Docs: https://opensearch.org/
Main PID: 1089 (java)
Tasks: 112 (limit: 4915)
CGroup: /system.slice/opensearch.service
└─ 1089 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=U>
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:179)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:217)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.jav>
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Jun 05 00:00:04 mygraylog systemd-entrypoint[1089]: at java.base/java.lang.Thread.run(Thread.java:833)
lines 1-19/19 (END)

Have you tried to create a new widget or is this the default? if so do you get the same results?

Same result if I use the default widget, an imported from Marketplace and if I create a new one.

How did you configure Opensearch && Graylog? Are you bale to show the configuration files for each?

Below I have provided the uncommented sections of the config files:

/etc/opensearch/opensearch.yml:
cluster.name: mygraylog.dom.name
path.data: /data/opensearch
path.logs: /var/log/opensearch

/etc/graylog/server/server.conf:
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret =
root_username =
root_password_sha2 =
bin_dir = /usr/share/graylog-server/bin
data_dir = /data/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.10.21:9000
http_enable_tls = true
http_tls_cert_file = //mygraylog.dom.name.fullchained.crt
http_tls_key_file = //mygraylog.dom.name.pkcs8.key
stream_aware_field_types=false
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /data/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 25gb
message_journal_flush_age = 5m
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000

What documentation did you use to install Graylog/OS/MongoDb?

https://go2docs.graylog.org/5-0/downloading_and_installing_graylog/suse_installation.htm

Best regards
/Flemming

fvr_flho · June 6, 2023, 7:32am

Hi

By the way, I have installed the following from marketplace:

https://community.graylog.org/t/windows-dns-for-graylog/22999

And according to the documentation for this, I have made the following change:

curl -XPUT localhost:9200/_index_template/graylog -H ‘Content-Type: application/json’ -d ’
{
“index_patterns”: [“graylog*”],
“template”: {
“settings”: {
“index.refresh_interval”: “30s”
},
“mappings”: {
“properties”: {
“ThreadID”: {
“type”: “keyword”
}
}
}
}
}’

Best regards
/Flemming

fvr_flho · June 6, 2023, 9:53am

Hi

Just as information - tried to remove the setting made previously with:

curl -XDELETE localhost:9200/_index_template/graylog -H ‘Content-Type: application/json’

Didn’t help - still no possibility to use Graylog at all

gsmith · June 7, 2023, 2:06am

Hey @fvr_flho

Thanks for the info. First thing I would suggest is not use the plugin and see if you can create your own widget.
I spotted a couple things.Your graylog config file does not seem correct.

Graylog configuration file.

Your configuration.

http_bind_address = 192.168.10.21:9000
http_enable_tls = true
http_tls_cert_file = //mygraylog.dom.name.fullchained.crt
http_tls_key_file = //mygraylog.dom.name.pkcs8.key

My configuration

[root@graylog graylog_user]# cat /etc/graylog/server/server.conf  | egrep -v "^\s*(#|$)"
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = epOqmLi7r7CdZxl76QOQxr8bRUP
root_password_sha2 = 5e884898da28047151d0e56f8dc6
root_email = "greg.smith@domain.com"
root_timezone = America/Chicago
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.100:9000
http_publish_uri = https://graylog.doamin.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Opensearch Configuration

If you using localhost in connecting Graylog to Opensearch. Here is basic configurations

cluster.name: graylog
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: localhost {and/or 127.0.0.1}
http.port: 9200
action.auto_create_index: false
discovery.type: single-node
bootstrap.memory_lock: true
plugins.security.disabled: true
plugins.security.system_indices.enabled: false

If you can run Graylog without certificates (i.e., HTTP) then either its your cretificates or configurations made. Dont forget to check firewall or Apparmor/Selinux , just in case.

If you followed this documentation for certificates.

Using HTTPS

Then you should have these two certificates…

hope that helps

fvr_flho · June 7, 2023, 8:08am

Hi @gsmith

I think the only difference between us, is the naming of the files… my .crt & .key is the same as .pem in your setup (old habit from my side, I prefer the naming on order to differ between the key and the cert)

I have tried without SSL, and the problem remains - So I suspect the upgrade has messed something up in OepnSearch… maybe I should open a bug

Best regards
/Flemming

system · June 21, 2023, 8:09am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error : While retrieving data for this widget Graylog Central (peer support)	4	5472	May 4, 2021
View log error,help me please Graylog Central (peer support)	11	926	May 12, 2021
Dashboards update Slowly Graylog Central (peer support)	2	681	April 15, 2021
Stability problem Graylog Central (peer support) sidecar , filebeat-linux , clfblx	17	2678	August 29, 2017
Graylog - No received messages Graylog Central (peer support)	8	733	June 18, 2021

Problem retrieving data for widget

Related topics