Sorry @lkenetadmin i dont mean to hijack your post, but we too just discovered, today, that we could not see logs in the “Search” page in one of our graylog server.
current graylog setup:
rsyslog port 514 receives all logs and sends them to the respective graylog inputs which in turn sends it to elasticsearch
( rsyslog, elasticsearch, graylog are all in the same server )
- Yes, rsyslog is receiving log
- yes, rsyslog is sending logs to graylog inputs - i can see the docs count in elasticsearch increasing
- yes, the time stamp is the same timezone (utc), ive even deleted all users just in case. (had this issue before)
- i can even use the “Show the Elasticsearch query” query graylog web provides in the search page and i can see all logs in the lates index
graylog2_55
with the current timestamp
logger "this is dave testing"
curl -X GET 'http://localhost:9200/graylog2_55/_search?q=message:this' | jq '.hits.hits[] | ._source.source, ._source.message, ._source.timestamp'
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:23:34.695"
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:37:04.695"
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:37:04.695"
date
Wed Apr 18 13:36:47 UTC 2018
curl -X GET 'http://localhost:9200/graylog2_55/message/_search?pretty=true' -d @elasticsearch_query.json | jq '.hits.hits[] | ._source.source, ._source.message, ._source.timestamp'
"KOR-RSD-2N-AP-4"
"*May 16 06:45:10.001 UTC: %DOT11-4-MAXRETRIES: Packet to client d02b.20f2.7be6 reached max retries, removing the client"
"2018-04-18 13:39:26.695"
"THA-CNX1A-A2-AP-1"
"*Sep 21 09:26:05.385 UTC: %DOT11-6-DISASSOC: Interface Dot11Radio0, Deauthenticating Station 5c8d.4e1b.f85e Reason: Sending station has left the BSS"
"2018-04-18 13:39:26.695"
As fas as Graylog inputs are concerened, they are all “Syslog - UDP”
Versions:
rpm -qa "graylog-server"
graylog-server-2.4.3-1.noarch
curl -X GET 'http://localhost:9200'
{
"name" : "GsFahYH",
"cluster_name" : "graylog2",
"cluster_uuid" : "8b_PMxV0TnOzHhVxd9MZ1g",
"version" : {
"number" : "5.6.8",
"build_hash" : "688ecce",
"build_date" : "2018-02-16T16:46:30.010Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
CentOS Linux release 7.4.1708 (Core)
3.10.0-693.21.1.el7.x86_64
I can go back in the Search page up to 30 days and see logs…
The last “visible” logs from the gui Search page stop at 2018-04-02.
On that date, we did do our monthly routine OS updates with a reboot, so could be related…maybe?
yum history info 189
Loaded plugins: fastestmirror
Transaction ID : 189
Begin time : Mon Apr 2 12:44:58 2018
Begin rpmdb : 645:c7aacdfe397d791c5ebd78e05efaab126976cb87
End time : 12:47:15 2018 (137 seconds)
End rpmdb : 645:6cc2a9268105bf5b029b5c962fa17409d7877371
User : root <root>
Return-Code : Success
Command Line : update -y
Transaction performed with:
Installed rpm-4.11.3-25.el7.x86_64 @base
Installed yum-3.4.3-154.el7.centos.1.noarch @updates
Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
Packages Altered:
Updated cpp-4.8.5-16.el7_4.1.x86_64 @updates
Update 4.8.5-16.el7_4.2.x86_64 @updates
Updated dhclient-12:4.2.5-58.el7.centos.1.x86_64 @updates
Update 12:4.2.5-58.el7.centos.3.x86_64 @updates
Updated dhcp-common-12:4.2.5-58.el7.centos.1.x86_64 @updates
Update 12:4.2.5-58.el7.centos.3.x86_64 @updates
Updated dhcp-libs-12:4.2.5-58.el7.centos.1.x86_64 @updates
Update 12:4.2.5-58.el7.centos.3.x86_64 @updates
Updated gcc-4.8.5-16.el7_4.1.x86_64 @updates
Update 4.8.5-16.el7_4.2.x86_64 @updates
Updated iptables-1.4.21-18.2.el7_4.x86_64 @updates
Update 1.4.21-18.3.el7_4.x86_64 @updates
Erase kernel-3.10.0-693.11.6.el7.x86_64 @updates
Install kernel-3.10.0-693.21.1.el7.x86_64 @updates
Updated kernel-headers-3.10.0-693.17.1.el7.x86_64 @updates
Update 3.10.0-693.21.1.el7.x86_64 @updates
Updated kernel-tools-3.10.0-693.17.1.el7.x86_64 @updates
Update 3.10.0-693.21.1.el7.x86_64 @updates
Updated kernel-tools-libs-3.10.0-693.17.1.el7.x86_64 @updates
Update 3.10.0-693.21.1.el7.x86_64 @updates
Updated libgcc-4.8.5-16.el7_4.1.x86_64 @updates
Update 4.8.5-16.el7_4.2.x86_64 @updates
Updated libgomp-4.8.5-16.el7_4.1.x86_64 @updates
Update 4.8.5-16.el7_4.2.x86_64 @updates
Updated libstdc++-4.8.5-16.el7_4.1.x86_64 @updates
Update 4.8.5-16.el7_4.2.x86_64 @updates
Updated libteam-1.25-5.el7.x86_64 @base
Update 1.25-6.el7_4.3.x86_64 @updates
Updated libtevent-0.9.31-1.el7.x86_64 @base
Update 0.9.31-2.el7_4.x86_64 @updates
Updated python-perf-3.10.0-693.17.1.el7.x86_64 @updates
Update 3.10.0-693.21.1.el7.x86_64 @updates
Updated ruby-2.0.0.648-30.el7.x86_64 @base
Update 2.0.0.648-33.el7_4.x86_64 @updates
Updated ruby-devel-2.0.0.648-30.el7.x86_64 @base
Update 2.0.0.648-33.el7_4.x86_64 @updates
Updated ruby-irb-2.0.0.648-30.el7.noarch @base
Update 2.0.0.648-33.el7_4.noarch @updates
Updated ruby-libs-2.0.0.648-30.el7.x86_64 @base
Update 2.0.0.648-33.el7_4.x86_64 @updates
Updated rubygem-bigdecimal-1.2.0-30.el7.x86_64 @base
Update 1.2.0-33.el7_4.x86_64 @updates
Updated rubygem-io-console-0.4.2-30.el7.x86_64 @base
Update 0.4.2-33.el7_4.x86_64 @updates
Updated rubygem-json-1.7.7-30.el7.x86_64 @base
Update 1.7.7-33.el7_4.x86_64 @updates
Updated rubygem-psych-2.0.0-30.el7.x86_64 @base
Update 2.0.0-33.el7_4.x86_64 @updates
Updated rubygem-rdoc-4.0.0-30.el7.noarch @base
Update 4.0.0-33.el7_4.noarch @updates
Updated rubygems-2.0.14.1-30.el7.noarch @base
Update 2.0.14.1-33.el7_4.noarch @updates
Updated selinux-policy-3.13.1-166.el7_4.7.noarch @updates
Update 3.13.1-166.el7_4.9.noarch @updates
Updated selinux-policy-targeted-3.13.1-166.el7_4.7.noarch @updates
Update 3.13.1-166.el7_4.9.noarch @updates
Updated systemd-219-42.el7_4.7.x86_64 @updates
Update 219-42.el7_4.10.x86_64 @updates
Updated systemd-libs-219-42.el7_4.7.x86_64 @updates
Update 219-42.el7_4.10.x86_64 @updates
Updated systemd-python-219-42.el7_4.7.x86_64 @updates
Update 219-42.el7_4.10.x86_64 @updates
Updated systemd-sysv-219-42.el7_4.7.x86_64 @updates
Update 219-42.el7_4.10.x86_64 @updates
Updated teamd-1.25-5.el7.x86_64 @base
Update 1.25-6.el7_4.3.x86_64 @updates
Updated tzdata-2018c-1.el7.noarch @updates
Update 2018d-1.el7.noarch @updates
Updated tzdata-java-2018c-1.el7.noarch @updates
Update 2018d-1.el7.noarch @updates
history info
Im at a lost as to where else to look I’ve verified the configs as well…
Sorry again @lkenetadmin ! i just happen to see this post in the “Latest” page and thought that maybe i was not the only one…plus we keep things in one location i guess…
I will gladly make another post if that is what is preferred!
thank you,
dave