Use both Graylog and Elastic Stack


(Arcturus999) #1

Hey guys, currently the compagny I work in has both graylog and elk. The architecture looks like this :
Filebeat --> Logstash --> Graylog --> Elasticsearch <— Kibana (and other stuff). The thing is having Graylog controlling the indices’ creation sorta bothers me, as I wish to be able to name my indices using custom names ( per app / μservice ) among other things. What I wish to accomplish is an architecture looking like this :
Filebeat --> Logstash --> Elasticsearch <== [ GrayLog | Kibana ] making ES the central node .
I don’t know if it’s possible to achieve that ( Is it possible for graylog to “read” from an existing ES node ? ).
I’m open to any suggestion :slight_smile:


(Jan Doberstein) #2

your wish architecture is not possible.

But you know that you can have multiple indices in Graylog? Those can have different retention strategys and names.

Just that you have it in mind - using Kibana to access the Graylog created indices/data in Elasticsearch bypass every security setting you might have in Graylog (different users are only able to see small parts of the messages that are ingested).


(Arcturus999) #3

Thanks for the reply :slight_smile: … Mmm I see, so there is no way to write down indices on the ES and force graylog to read from it … Is there any way to make ELK and Graylog coexist ? I’m thinking of separating logstash’s output into two, a regular ES one and a Gelf for Graylog but I’m afraid that this will duplicate the indices’ creation thus consuming too much memory.


(Jan Doberstein) #4

if you split messages with logstash - send one message to Graylog and Elasticsearch direct you will have that saved twice.

Why did you want to have that kind of setup? Maybe we should look from that end.


(Arcturus999) #5

I unfortunately didn’t chose this setup :\ , it was already there when I arrived ( for my internship ). The fact is they had many problems using both ELK and GrayLog as the ops prefer using Graylog and the dev like having a real-time log viewer using a forked project called Logtrail for Kibana which enables a sort of shell for viewing logs similar to using tail.
I tried looking into other solutions such as removing Kibana from the stack, since it is not used for much apart from logtrail, especially when the whole analysis is done on Graylog, but the dev really want a real time log viewer ( I don’t know if this can be done on graylog ).


(Jan Doberstein) #6

in the current stable version you can follow a search and keep it self reloading all X sec/min - I guess that something like this is what they want.

Should the developers do not like the way the logfiles are displayed, they can use the API to get the streams content.

With the upcoming version 3.0 the look and feel of the search - and the way you can customize the search view - will change and might be then more what your devs are looking for.


(Arcturus999) #7

Is there an API that mimics Logtrail on Graylog ( Sorry for all those questions, I’m quite new to Graylog ) ?


(Jan Doberstein) #8

as I do not known how logtrail is working I can’t answer that question.


(Arcturus999) #9

It’s basically a shell where you can tail your logs. But I guess as you said automatically reloading the search every 1 second does the job just fine :slight_smile: . That would then leave me with a Kibana-less Elastic stack. At least it’s much better than having duplicated indices on two separate Elasticsearch nodes. Tho I can’t help but to think that reading existing ES indices would be quite a nice feature on Graylog ( maybe for hard coded apps which directly write on ES ? ).


(Jan Doberstein) #10

feel free to create a feature issue over at github - if this does not exists already.

As Graylog needs some additional (meta) information saved with the data to be able to make the access seperation possible it is not very likely that we will include this.


(Arcturus999) #11

Alrightyyy, Thanks a lot for your help !


(system) #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.