Logs not showing in web ui


#1

Hi everyone,

I’m having problems with my graylog server. There are no logs being shown as seen in this picture

I did not suspect the ntp of our server since it’s I already configured it already with the correct ntp and time
Time%20config

Plus it’s processing the messages
Metrics

Any help would be great :slight_smile:


(Jochen) #2

Try extending the time range of your search a few hours (let’s say 8 hours) into the future.


#3

Tried your suggestion but still nothing


(Jochen) #4

What type of input are you using (including the complete configuration) and what kind of messages are being sent to that input?
What’s in the logs of your Graylog node?


#5

Currently there are two inputs that are running and both of them are showing the logs but is it processed to elasticsearch.



#6

Is there any problems with our graylog server?


(Jochen) #7

That’s for you to find out by checking the logs of your Graylog and Elasticsearch nodes.
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Dave) #8

Sorry @lkenetadmin i dont mean to hijack your post, but we too just discovered, today, that we could not see logs in the “Search” page in one of our graylog server.

current graylog setup:

rsyslog port 514 receives all logs and sends them to the respective graylog inputs which in turn sends it to elasticsearch
( rsyslog, elasticsearch, graylog are all in the same server )

  • Yes, rsyslog is receiving log
  • yes, rsyslog is sending logs to graylog inputs - i can see the docs count in elasticsearch increasing
  • yes, the time stamp is the same timezone (utc), ive even deleted all users just in case. (had this issue before)
  • i can even use the “Show the Elasticsearch query” query graylog web provides in the search page and i can see all logs in the lates index graylog2_55 with the current timestamp
logger "this is dave testing"

curl -X GET 'http://localhost:9200/graylog2_55/_search?q=message:this' | jq  '.hits.hits[] | ._source.source, ._source.message, ._source.timestamp' 
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:23:34.695"
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:37:04.695"
"jpn-off-1se-ecr-1"
"this is dave testing"
"2018-04-18 13:37:04.695"
date
Wed Apr 18 13:36:47 UTC 2018
curl -X GET 'http://localhost:9200/graylog2_55/message/_search?pretty=true' -d @elasticsearch_query.json | jq  '.hits.hits[] | ._source.source, ._source.message, ._source.timestamp' 
"KOR-RSD-2N-AP-4"
"*May 16 06:45:10.001 UTC: %DOT11-4-MAXRETRIES: Packet to client d02b.20f2.7be6 reached max retries, removing the client"
"2018-04-18 13:39:26.695"
"THA-CNX1A-A2-AP-1"
"*Sep 21 09:26:05.385 UTC: %DOT11-6-DISASSOC: Interface Dot11Radio0, Deauthenticating Station 5c8d.4e1b.f85e Reason: Sending station has left the BSS"
"2018-04-18 13:39:26.695"

As fas as Graylog inputs are concerened, they are all “Syslog - UDP”

Versions:

rpm -qa "graylog-server"
graylog-server-2.4.3-1.noarch
curl -X GET 'http://localhost:9200'
{
  "name" : "GsFahYH",
  "cluster_name" : "graylog2",
  "cluster_uuid" : "8b_PMxV0TnOzHhVxd9MZ1g",
  "version" : {
    "number" : "5.6.8",
    "build_hash" : "688ecce",
    "build_date" : "2018-02-16T16:46:30.010Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
CentOS Linux release 7.4.1708 (Core) 
3.10.0-693.21.1.el7.x86_64

I can go back in the Search page up to 30 days and see logs…
The last “visible” logs from the gui Search page stop at 2018-04-02.
On that date, we did do our monthly routine OS updates with a reboot, so could be related…maybe?

yum history info 189
Loaded plugins: fastestmirror
Transaction ID : 189
Begin time     : Mon Apr  2 12:44:58 2018
Begin rpmdb    : 645:c7aacdfe397d791c5ebd78e05efaab126976cb87
End time       :            12:47:15 2018 (137 seconds)
End rpmdb      : 645:6cc2a9268105bf5b029b5c962fa17409d7877371
User           : root <root>
Return-Code    : Success
Command Line   : update -y
Transaction performed with:
    Installed     rpm-4.11.3-25.el7.x86_64                      @base
    Installed     yum-3.4.3-154.el7.centos.1.noarch             @updates
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64       @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
Packages Altered:
    Updated cpp-4.8.5-16.el7_4.1.x86_64                       @updates
    Update      4.8.5-16.el7_4.2.x86_64                       @updates
    Updated dhclient-12:4.2.5-58.el7.centos.1.x86_64          @updates
    Update           12:4.2.5-58.el7.centos.3.x86_64          @updates
    Updated dhcp-common-12:4.2.5-58.el7.centos.1.x86_64       @updates
    Update              12:4.2.5-58.el7.centos.3.x86_64       @updates
    Updated dhcp-libs-12:4.2.5-58.el7.centos.1.x86_64         @updates
    Update            12:4.2.5-58.el7.centos.3.x86_64         @updates
    Updated gcc-4.8.5-16.el7_4.1.x86_64                       @updates
    Update      4.8.5-16.el7_4.2.x86_64                       @updates
    Updated iptables-1.4.21-18.2.el7_4.x86_64                 @updates
    Update           1.4.21-18.3.el7_4.x86_64                 @updates
    Erase   kernel-3.10.0-693.11.6.el7.x86_64                 @updates
    Install kernel-3.10.0-693.21.1.el7.x86_64                 @updates
    Updated kernel-headers-3.10.0-693.17.1.el7.x86_64         @updates
    Update                 3.10.0-693.21.1.el7.x86_64         @updates
    Updated kernel-tools-3.10.0-693.17.1.el7.x86_64           @updates
    Update               3.10.0-693.21.1.el7.x86_64           @updates
    Updated kernel-tools-libs-3.10.0-693.17.1.el7.x86_64      @updates
    Update                    3.10.0-693.21.1.el7.x86_64      @updates
    Updated libgcc-4.8.5-16.el7_4.1.x86_64                    @updates
    Update         4.8.5-16.el7_4.2.x86_64                    @updates
    Updated libgomp-4.8.5-16.el7_4.1.x86_64                   @updates
    Update          4.8.5-16.el7_4.2.x86_64                   @updates
    Updated libstdc++-4.8.5-16.el7_4.1.x86_64                 @updates
    Update            4.8.5-16.el7_4.2.x86_64                 @updates
    Updated libteam-1.25-5.el7.x86_64                         @base
    Update          1.25-6.el7_4.3.x86_64                     @updates
    Updated libtevent-0.9.31-1.el7.x86_64                     @base
    Update            0.9.31-2.el7_4.x86_64                   @updates
    Updated python-perf-3.10.0-693.17.1.el7.x86_64            @updates
    Update              3.10.0-693.21.1.el7.x86_64            @updates
    Updated ruby-2.0.0.648-30.el7.x86_64                      @base
    Update       2.0.0.648-33.el7_4.x86_64                    @updates
    Updated ruby-devel-2.0.0.648-30.el7.x86_64                @base
    Update             2.0.0.648-33.el7_4.x86_64              @updates
    Updated ruby-irb-2.0.0.648-30.el7.noarch                  @base
    Update           2.0.0.648-33.el7_4.noarch                @updates
    Updated ruby-libs-2.0.0.648-30.el7.x86_64                 @base
    Update            2.0.0.648-33.el7_4.x86_64               @updates
    Updated rubygem-bigdecimal-1.2.0-30.el7.x86_64            @base
    Update                     1.2.0-33.el7_4.x86_64          @updates
    Updated rubygem-io-console-0.4.2-30.el7.x86_64            @base
    Update                     0.4.2-33.el7_4.x86_64          @updates
    Updated rubygem-json-1.7.7-30.el7.x86_64                  @base
    Update               1.7.7-33.el7_4.x86_64                @updates
    Updated rubygem-psych-2.0.0-30.el7.x86_64                 @base
    Update                2.0.0-33.el7_4.x86_64               @updates
    Updated rubygem-rdoc-4.0.0-30.el7.noarch                  @base
    Update               4.0.0-33.el7_4.noarch                @updates
    Updated rubygems-2.0.14.1-30.el7.noarch                   @base
    Update           2.0.14.1-33.el7_4.noarch                 @updates
    Updated selinux-policy-3.13.1-166.el7_4.7.noarch          @updates
    Update                 3.13.1-166.el7_4.9.noarch          @updates
    Updated selinux-policy-targeted-3.13.1-166.el7_4.7.noarch @updates
    Update                          3.13.1-166.el7_4.9.noarch @updates
    Updated systemd-219-42.el7_4.7.x86_64                     @updates
    Update          219-42.el7_4.10.x86_64                    @updates
    Updated systemd-libs-219-42.el7_4.7.x86_64                @updates
    Update               219-42.el7_4.10.x86_64               @updates
    Updated systemd-python-219-42.el7_4.7.x86_64              @updates
    Update                 219-42.el7_4.10.x86_64             @updates
    Updated systemd-sysv-219-42.el7_4.7.x86_64                @updates
    Update               219-42.el7_4.10.x86_64               @updates
    Updated teamd-1.25-5.el7.x86_64                           @base
    Update        1.25-6.el7_4.3.x86_64                       @updates
    Updated tzdata-2018c-1.el7.noarch                         @updates
    Update         2018d-1.el7.noarch                         @updates
    Updated tzdata-java-2018c-1.el7.noarch                    @updates
    Update              2018d-1.el7.noarch                    @updates
history info

Im at a lost as to where else to look :frowning_face: I’ve verified the configs as well…

Sorry again @lkenetadmin ! i just happen to see this post in the “Latest” page and thought that maybe i was not the only one…plus we keep things in one location i guess…

I will gladly make another post if that is what is preferred! :smile:

thank you,
dave


(Dave) #9

My personal issue has been solved!! :smile:

I could not understand why starting 2018-04-02 the logs stop show visibly in the UI.
I thought of perhaps renaming that index ‘graylog2-55’ so that graylog-server can create a new index and see what would happen.

As i looked at
System>Indices>Default Index set>graylog2_55

I noticed this “No index range available”:

I compared it with the current index in the other Graylog-Servers and they said “Range re-calculated 3 hours ago”:

So i recalculated the index ranges and boom! I could see logs in the normal Search page.

@lkenetadmin hopefully you were able to find your solution

thanks!
dave


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.