1. Describe your incident:
I have a running Graylog that has been collecting syslog data from firewalls (SonicWALL) and Windows hosts (via sidecar) for months.
This week I wanted to add our Fortinet load balancer (FortiADC) to the mix. I create an input for syslog. When I click on “show messages” for the input, I can see messages and also I grab the gl2_source_input. The first thing I notice here is that the message count graph is empty even though there are messages in the table.
My steps for the previous devices was to create an Index, and then a stream. Index is pretty much default. When I go to the stream config I set it up and point it at the Index I created. I then create a stream rule for gl2_source_input equal to the value found when viewing the messages. If I click “load message” from the Source it will show as matching the stream rule. So I create the stream.
But then when I try to search on the stream it tells me " * Elasticsearch exception [type=index_not_found_exception, reason=no such index ]."
If I go run the curl command to list the indexes, sure enough it’s not there.
I’m not sure what is different here. I even tried plaintext/UDP and get he same results. Also tried TCP for both syslog and plain text without success.
I did look up some Fortinet posts and articles. They are all focused on the FortiGate and FortiOS systems. This is not that. It’s a load balancer. And none of the recommendations for the FortiGate/OS helped at all.
2. Describe your environment:
OS Information: Ubuntu 22.04.2 LTS
Package Version: Graylog Open 5.1.7-1
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
Tried different input types. Syslog UDP and TCP, plain text UDP and TCP. Also re-built the input, index, and stream multiple times when changing any setting.
4. How can the community help?
I’m not sure where to look to see what the problem might be. I’m presuming it has something to do with the incoming data. But I’m not sure what it might be.
I tried searching for something that was unique to the messages (the device’s serial number). This turned up no results.
The weird thing (to me at least) is that if I click Show Received Messages from the Input screen it brings up the messages with gl2_source_input and the id as the search term. But if I copy that and open a new search, it comes up blank.
I went to look at that, but didn’t find any new data. Poked around and found that the graylog server had run out of space. Resolved that and got back to it. Still no new incoming data into the input. I then went back to zero and re-created the input. But it’s still showing no data. Packet captures at both ends show data being sent and received. I’m lost now.
The time stamps all look fine. But I’m still having the issues with the data not showing up in search. If I look at the data from the “show messages” on the input it does show the messages. And that they’re routed to the correct Stream.
But if I search by the stream or the gl2-source ID the results are empty.
Now this is strange. After going to some meetings and then coming back, I noticed something strange.
If I add the Stream to the search and then set the Time Range to anything above 4 hours, I will get results. Including the most current messages. I even experimented with different times. If I set it for say 240 or 241 minutes I’ll see messages. If I set it to 235, nothing. Maybe the indexer is just running really slowly, and all this time I just haven’t been patient enough, or thought to expand my time range.
You have a timezone issue is my guess. Compare that time in the actual message to the time on the left when you expand the message, they will be off by 4 hours probably. Check this blog post to see if anything helps https://graylog.org/post/time-zones-a-loggers-worst-nightmare/
