Search is empty even elasticsearch has data in index

[satishdotpatel]

I have just setup new graylog and playing with it and found my clients sending logs to graylog but its not showing in search or anywhere, if i query data in elasticsearch i can see all of my data.

• I am using UDP 1514 input for Cisco syslog

• Gaylog 2.4

• ES - 5.x

• MongoDB 3.6

  [root@graylog ~]# curl ‘localhost:9200/_cat/indices?v’
  health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
  green open graylog_1 -ZESHG13SiGV9uY5jBNqRg 4 0 22 0 151.9kb 151.9kb
  close graylog_0 1BkG2DhhSNym4Y1owDtTUg

I can see all my logs in following command

[root@graylog ~]# curl -H 'Content-Type: application/json' -X GET http://localhost:9200/graylog_1/_search?pretty
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 4,
    "successful" : 4,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 22,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "graylog_1",
        "_type" : "message",
        "_id" : "8b987d10-d67e-11e8-bb7b-6c3be5a99590",
        "_score" : 1.0,
        "_source" : {
          "level" : 5,
          "streams" : [
            "000000000000000000000001"
          ],
          "source" : "unknown",
          "message" : "%ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/1 is down (Administratively down)",
          "gl2_source_input" : "5bce8baf370f98103d87fe4a",
          "sequence_number" : 0,
          "full_message" : "<181>: 2018 Oct 23 01:56:24 UTC: %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/1 is down (Administratively down)",
          "gl2_source_node" : "cfa5b46c-2c4a-4904-9905-6e3e8a1386ed",
          "facility" : "local6",
          "timestamp" : "2018-10-23 01:56:24.000"
        }
      },
      {

....
....

My Switches using following EST timestamp for logging and i have changed root_timezone = EST in graylog but still no luck, i have increase my search to 1 year but still no data.

Side note: I am running input on 1514 because graylog not allowing me to open port on 514.
also my cisco switch default send logs on 514 so i have setup iptables to REDIRECT my all logs from 514 --> 1514 ( do you think this could be the issue?)

iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514
iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514

What could be wrong here?

[Ponet (Jesse Hills)]

Hi @satishdotpatel

If you go to the System/Overview tab within the Graylog Web Interface and scroll down, you should see three values for Time.

One for the currently logged in user, One for Graylog and another for the Server that Graylog is running on.

Are those times the same/what you’re expecting them to be?

— EDIT —

Are you not able to change the port that your Cisco devices send the syslog messages to?

Thanks

[satishdotpatel]

Those time is in EST and that is matching with my Cisco switch time.

Yes i can’t change logging port on Cisco switches because Cisco nexus don’t allow you to change destination port, that feature is only for IOS base switches.

Is there a way i can force graylog to listen on 514 port that would be awesome!

[Ponet (Jesse Hills)]

Hi again,

So, the time configuration you posted above matches the time on your Cisco switches as well, right?

If you run tcpdump filtering on packets destined for Port 1514 or 514 on your graylog server, can you see data coming into the box from your switches?

[satishdotpatel]

Yes timestamp is matching on switches and graylog server (both on EST)

Yes they are coming to 514 and 1514 port and they are going in elasticsearch, i can see them as i posted above output you can see cisco syslog messages.

I am able to browse all logs using curl command against elasticsearch index so i am 100% sure its working, my Document count is rising too

Only graylog not showing anything in search

[Ponet (Jesse Hills)]

Can you see any recent messages through Elasticsearch?

The timestamp of the example you posted in the first message is nearly 2AM

[satishdotpatel]

I think i messed up i was trying to delete all mesg from index graylog_1 but by mistake deleted index itself.

I had two index before graylog_1 and graylog_0 and all mesg were going in grayog_1 but after delete graylog_1 mesg not going in graylog_0

How do i get my graylog_1 index back or is there a good way to re-create index?

[Ponet (Jesse Hills)]

I’m not too familiar with dealing with the index sets since I’ve never really had to do much with them, thankfully mine have just worked…

The Graylog documentation may give you some ideas as to what you should do.

[satishdotpatel]

I am seeing this error in graylog server.log

2018-10-23T12:35:12.519-04:00 WARN  [Messages] Failed to index message: index=<graylog_0> id=<9cc2adf0-d6e1-11e8-9779-6c3be5a99590> error=<{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/8/index write (api)];"}>
2018-10-23T12:35:12.520-04:00 ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2018-10-23T12:35:13.517-04:00 WARN  [Messages] Failed to index message: index=<graylog_0> id=<9d5b1d60-d6e1-11e8-9779-6c3be5a99590> error=<{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/8/index write (api)];"}>
2018-10-23T12:35:13.518-04:00 ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

[satishdotpatel]

Finally after rorated index it works!!! damn it!!