Good morning!
I am inserting information from the console to an Elasticsearch index created in Graylog 4.3.15, the issue is that I cannot see the information in the Graylog search but the inserted data is seen in the index menu, however I do the curl In elastic the information is searchable, why does this way of working?
Thanks greetings!
Hey @ferdagon
Give us some details about how your inserting information from the console? Is this through an INPUT?
Can you show us what kind of search are you executing?
Good morning,
I put the information with a script that performs several checks and finally the function that reindexes one index by copying another is the one you can see below:
Additionally, I check the console queries like this, pointing to the index where I inserted the information with this query:
curl -XGET “http://elastic/index/_search?pretty=true”
The indexset where the indexes are inserted…:
Searches on that stream that has the indexset backed up with the indexes where the information was inserted, there is information from less than a month ago, searching through the console I see the ones but in graylog they are not seen
Thanks greetings!
Hey
Correct me if I’m wrong,
You coped one index set, lets say index-1 into lets say,. index-2. Through a curl command you can see the index set BUT you can not see the copied logs/message on Graylog Web UI. If so have you tried rotating the index sets?
What I think is going on is Graylog is not picking it up. I would look through the logs of ES/Graylog and MongoDb for errors or anything that would pertain to this issue.
What I would try is executing the same thing from your script BUT using the Graylog Web UI. instead.
If that is a success then you know where the issue is.
Im not sure i’m 100% following what your doing, BUT Graylog doesn’t just automatically read everyrhing that is in elastic, it uses elastic/OS as the backend, but it’s not a viewer for those data sets. It expects that it wrote the information there so it will often store needed metadata in mongo with pointers etc, so you can’t necessarily just add things to the database and it will pick it up.
Good morning,
First of all, thank you for the help, so even if you have the index previously created in graylog and later insert data into it, but from the console, graylog is not able to read that data? Is there no way to tell it to read the data by reindexing the index/indexset or something like that? It seems strange to me that graylog creating the index, seeing its metadata and so on correctly and testing it from the graylog API, is not able to read data correctly inserted into that elastic index unless the messages are inserted passing the entire data flow from graylog (inputs->…Index)
Thanks greetings!
Hello again, as you can see Graylog can see the messages, they are in the elastic managed by graylog, but it does not load them unless it looks for a specific message id, is there a way for it to read everything?
Thanks greetings!
So thats definitly getting far away from what is documented, but my guess would be that you are missing some fields that are required stream, source, message, etc. I dont know that for sure, but thats my first guess.
Hey @ferdagon
Yes, with Graylog you need to send the data to Graylog’s INPUT. If you try to insert data using curl command you will have issue.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
