Elasticsearch input


(Gianluca Valentini) #1

Hi,
I have a new ElasticSearch and I need to send data from Elastic to Graylog.

Is there a simple way to connect them?
Thanks
Gianluca


#2

Do you want to send the Elasticsearch’s log to graylog?
In this case you can use rsyslog to read elasticsearch’s log, remove javas new line characters, and send it to graylog via syslog protocol.

important lines:

#remove local3.none from messages log
*.info;mail.none;authpriv.none;cron.none;local3.none    /var/log/messages

$ModLoad imfile
input(type="imfile" file="/var/log/elasticsearch/asp_kod_elasticsearch.log" escapelf="on" readMode="2" tag="elasticsearch.log" facility="local3")

*.* @@IP:514

(Gianluca Valentini) #3

Thanks for your help.

Currently I have an elastic search with a lot of data and I need to move them using graylog

So, I’d like to know if there is a way or an input that I can use to connect it.
Is there in your opinion a simple way to perform this operation?

Thanks
Gianluca


(Tess) #4

I’m still a bit baffled…

Do you want to “move the data using Graylog” as you say, or do you want to “move the data into Graylog”? And if you have to move it somewhere, where does it need to go and why?

You do realize that Graylog’s storage backend is ElasticSearch, right?


(Gianluca Valentini) #5

:slight_smile: sure!!
A customer has his data stored on an elastic search.
Using graylog and a kafka output, I need to move the customer data to kafka.
I hope I have explained better my use case and my need.

So I need to understand if there is a simple way to extract data from customer elastic search


#6

not so…

So you got an elastic database full with graylog’s indices?
And you would like to open this data with your graylog?

The graylog won’t process stored messages again, so you can’t send it to output via graylog.

Maybe you need elasticdump, after that you can do anything with the raw data.


(Ben van Staveren) #7

Okay a little bit more detail would be nice: what data does the customer have? Graylog indices? Or output from, say, Logstash? In the former case I don’t see what you are trying to do, in the latter case, you can use logstash with an elasticsearch input and a gelf output to re-submit messages to a Graylog instance.


(Gianluca Valentini) #8

we are talking about the second case.
Thanks a lot

Can you give me an example how to implement this scenario?
Can you suggest me a guide to use to implement the elastic search input?
Is required an agent installation (like graylog file best for example) in customer environment where elastic is present?


(Ben van Staveren) #9

Well, we have the same issue - sort of - we migrated from Filebeat -> Logstash -> Elasticsearch to using Graylog, and our “old” data still exists on a separate Elasticsearch cluster. What we’re doing now is using Logstash with an Elasticsearch input (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html) to read a day at a time (we used daily indices) and submit that via a Gelf output (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-gelf.html) to Graylog where it’s processed through a pipeline that parses the input and sets the timestamp to the “proper” time.

I can’t give you exact implementation details, that depends on the data, the setup they have, how much data there is, how much time you have, and so on and so forth. We also use a huge wad of Perl scripts to orchestrate this stuff, so unfortunately there is no “one size fits all” kind of solution.


(Gianluca Valentini) #10

I understand perfectly.
First of all thank you for answering me and giving the right instructions to address my scenario.
At first sight, it seems that this operation will require a lot of time both for study and implementation.
Thanks again


(Tess) #11

Good luck Gianluca! Sounds like an exciting project :slight_smile:

Thanks for sharing your experience Ben.


(Ben van Staveren) #12

Oh it’ll take a bit of time, unfortunately (well, in our case) logstash isn’t super fast due to the fact our “old” cluster got downsized significantly, so it’s taking time to actually perform the entire thing, but it is doable. It may not be the best way, but it works for our case :slight_smile:

Let us know how it goes! :slight_smile:


(system) #13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.