Hi,
I have a new ElasticSearch and I need to send data from Elastic to Graylog.
Is there a simple way to connect them?
Thanks
Gianluca
Do you want to send the Elasticsearchâs log to graylog?
In this case you can use rsyslog to read elasticsearchâs log, remove javas new line characters, and send it to graylog via syslog protocol.
important lines:
#remove local3.none from messages log
*.info;mail.none;authpriv.none;cron.none;local3.none /var/log/messages
$ModLoad imfile
input(type="imfile" file="/var/log/elasticsearch/asp_kod_elasticsearch.log" escapelf="on" readMode="2" tag="elasticsearch.log" facility="local3")
*.* @@IP:514
Thanks for your help.
Currently I have an elastic search with a lot of data and I need to move them using graylog
So, Iâd like to know if there is a way or an input that I can use to connect it.
Is there in your opinion a simple way to perform this operation?
Thanks
Gianluca
Iâm still a bit baffledâŠ
Do you want to âmove the data using Graylogâ as you say, or do you want to âmove the data into Graylogâ? And if you have to move it somewhere, where does it need to go and why?
You do realize that Graylogâs storage backend is ElasticSearch, right?
sure!!
A customer has his data stored on an elastic search.
Using graylog and a kafka output, I need to move the customer data to kafka.
I hope I have explained better my use case and my need.
So I need to understand if there is a simple way to extract data from customer elastic search
not soâŠ
So you got an elastic database full with graylogâs indices?
And you would like to open this data with your graylog?
The graylog wonât process stored messages again, so you canât send it to output via graylog.
Maybe you need elasticdump, after that you can do anything with the raw data.
Okay a little bit more detail would be nice: what data does the customer have? Graylog indices? Or output from, say, Logstash? In the former case I donât see what you are trying to do, in the latter case, you can use logstash with an elasticsearch input and a gelf output to re-submit messages to a Graylog instance.
we are talking about the second case.
Thanks a lot
Can you give me an example how to implement this scenario?
Can you suggest me a guide to use to implement the elastic search input?
Is required an agent installation (like graylog file best for example) in customer environment where elastic is present?
Well, we have the same issue - sort of - we migrated from Filebeat -> Logstash -> Elasticsearch to using Graylog, and our âoldâ data still exists on a separate Elasticsearch cluster. What weâre doing now is using Logstash with an Elasticsearch input (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html) to read a day at a time (we used daily indices) and submit that via a Gelf output (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-gelf.html) to Graylog where itâs processed through a pipeline that parses the input and sets the timestamp to the âproperâ time.
I canât give you exact implementation details, that depends on the data, the setup they have, how much data there is, how much time you have, and so on and so forth. We also use a huge wad of Perl scripts to orchestrate this stuff, so unfortunately there is no âone size fits allâ kind of solution.
2 Likes
I understand perfectly.
First of all thank you for answering me and giving the right instructions to address my scenario.
At first sight, it seems that this operation will require a lot of time both for study and implementation.
Thanks again
Good luck Gianluca! Sounds like an exciting project
Thanks for sharing your experience Ben.
Oh itâll take a bit of time, unfortunately (well, in our case) logstash isnât super fast due to the fact our âoldâ cluster got downsized significantly, so itâs taking time to actually perform the entire thing, but it is doable. It may not be the best way, but it works for our case
Let us know how it goes!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.