What would it take to have Graylog talk to Elasticsearch 8?

[nisow95612]

Hello,
I am sorta stuck between two hard places. On one side I have graylog that went with opensearch. On other side I have apps that went with elasticsearch.

So the question is: What would it take to make Graylog work with ES 8? I ask because my cheapest way forward may be paying someone to write a ES8 plugin for graylog But before I find money for it, can that work?

–

So, options range is like this

• Say goodbye to elasticsearch and all clients that went with it
• Pay someone to write a graylog plugin for ES8
• Do nothing and continue with (inferior) ES7.17 as long as possible
• Allow two different databases and search for everything twice
• Pay someone to write a elasticsearch plugin to provide ES7 API
  • or ES8 to ES7 API proxy that transforms incompatible calls
• Say goodbye to graylog and find different way to ingest my data

Obviously, this is unsustainable in log run, because ES and OS will grow apart more and more, but right now it’s not too big difference I think and it would allow me to everyone more time to think about the first/last option.

Hope its okay asking about elasticsearch here.

[Wine_Merchant]

Hey @nisow95612,

Consider that all Graylog development work will be focused on Opensearch moving forward even if you did get Graylog working with ES8 (perhaps an unrealistic undertaking) you would always be at risk that an update could break compatibility.

You could reindex the your data into an index version compatible with Opensearch and then migrate to Opensearch but if other apps require ES8 you would need to maintain two data stores.

[nisow95612]

Hello @Wine_Merchant ,
I edited question so it is more helpful to others with same situation.

That is what I hope to find out - is graylog plugin for ES8 realistic? I ask because I see ES7 and OS are graylog plugins, so it looks graylog is maybe prepared for third-party solutions like this. Yes, I know this is not sustainable in long run, but at least until ES7 plugin goes away, I think ES8 plugin can just pretend to be ES7 for rest of graylog. It is not a guarantee it won’t break, but in my situation I am OK with it.

Also, I want to just pospone the hard “keep graylog” v.s. “search everything twice” v.s. “keep elasticsearch” decision. I hope by time ES9 comes (or graylog drops ES7) it will be easier to see which path ES and OS go and which will work best for everyone.

[Wine_Merchant]

I’m not sure for how much longer the final supported version of Elastic (7.10.2) will remain supported and at what point the baseline becomes Opensearch. To me this seems unrealistic but in fairness I’ve not done much research into what incompatibilities running Graylog against Elastic 8 brings about, I was under the impression at least some of the reason Graylog went OS was due to ES introducing a licensing model.

In your scenario it would come down to monetary gain of have Graylog continue to work within your environment vs the cost of getting it there (assuming that is even possible and for ongoing development costs should other issues arise).

You could make use of the Graylog output function to forward logs but of course then you miss out on the parsing.

[nisow95612]

Right. I don’t know how long graylog keeps ES7. I remember ES6 was removed very recently with graylog 5, so I expect ES7 and OS1 (I remember API is same?) will stay longer time, but I can be wrong here - that is good point.

Well, my other options are “converting all ES clients to OS” or “converting all my graylog parsers to something else”. I see no way to avoid this decision in future, but right now ES and OS is almost same, so how can I choose what is better. Anyway, both options are very expensive and if I pay for wrong transition, that money will be pure waste. Searching everything twice is bad and reduces productivity in long run, so I don’t consider that.

Hm, your Graylog output idea can work, but how I get logs from output to ES? I think writing ES8 output plugin for graylog give same problem as writing ES8 database plugin for graylog? I can use graylogs GELF output, but then I must pay someone to write GELF to ES8 converter, right?
What you mean by “you miss out on the parsing”? Outputs send raw messages instead of processed ones?
But thank you for inspiration - If I pay for ES8 database plugin for graylog and new version of graylog stops compatibility, then I can install new graylog with GELF output, feeding older graylog version with GELF input and my ES8 plugin. At least until graylog stops GELF compatibility, but I think they wanted GELF to be standard so it should stay compatible, right?