Missing Data in Graylog, but in ElasticSearch

[arrmo (Russell Morris)]

Hi,

OK, pulling my hair out (and I don’t have enough to spare … ). I am sending data to Graylog, from rsyslog. I have used tcpdump, to make sure that the packets are getting there … they are. So then I query from ElasticSearch, using the API - example below, and data showing up,
curl -X GET "localhost:9200/graylog_0/_search?q=source:wap-attic&pretty"

I also used the API, to check the count - a bunch of records there,

curl -X GET "localhost:9200/graylog_0/_count?q=source:wap-attic&pretty"
{
  "count" : 69,
  "_shards" : {
    "total" : 4,
    "successful" : 4,
    "skipped" : 0,
    "failed" : 0
  }
}

But then I try exactly the same query (string) as above, in the Graylog GUI (selecting all time) => no records found! I wonder if it’s an issue with timezone (FYI, I’m sending from rsyslog in RSYSLOG_SyslogProtocol23Format … but have tried others, no joy). I say TZ, because if I look at one record from the API query - the time looks to be in the future, which makes no real sense. But could that be why nothing is showing up in the Graylog web interface?

Of course, how to fix the TZ also, if that is it

Thanks!

[arrmo (Russell Morris)]

OK, found it! If /etc/localtime is missing (which it is on this / some systems), the time is sent as (examples here),
2020-04-23T02:47:21.749833+00:00

Instead of,
2020-04-22T21:57:56.574015-05:00

But this should be accepted, no? They are the same thing, and the offset is shown. So why is Graylog ignoring the log entry? It is seen in ElasticSearch.

Thanks!

[jan (Jan Doberstein)]

@arrmo

the question is - how did you search in Graylog?

Graylog does honour the timezone of the local user - when you search in the last 5 minutes it will take the timezone from the user profil to search the last 5 minutes.

If your ingested messages are not from within this time range they will not show up.

[arrmo (Russell Morris)]

Hi,

I wasn’t limiting Graylog to any time range, rather just querying for that particular source, but without a time limitation. Similar to the curl query above, but in Graylog (vs. ES).

Thanks!