Recommended way to get the newest log entry for a pod

Graylog Central (peer support)
1

Hey all.

What’s the recommended way to get the newest/latest log entry for a pod via REST API?
I’ve tried using something like this:
https://graylog/api/search/universal/relative?range=0&query=k8s_pod%3A"podname"&limit=1&fields=message,timestamp
and it appeared to work fine, but this kind of request seems to cause pretty high load for the graylog - just a few such requests cause it to consume all available CPU for a long time.
Also after a while I started getting a cryptic “{“type”:“ApiError”,“message”:”“}” error message and an unhandled exception in the logs:

2021-08-09 15:58:06,202 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.NullPointerException: null
at org.graylog.storage.elasticsearch6.ScrollResultES6.(ScrollResultES6.java:90) ~[?:?]
at org.graylog.storage.elasticsearch6.ScrollResultES6$$FastClassByGuice$$69ff0573.newInstance() ~[?:?]

Am I doing something wrong? Is there any other, more optimal way to do it?
Also, how to debug this exception? I found several topics with similar exception, but none of them were answered :confused:

2

Hello && Welcome

I might be able to help.

Searches are performed by Elasticsearch, and if you don’t know already ES uses up CPU. This would depend on how far back your searching (i.e., minutes, hours, days, etc…) and what resource you provided for Graylog server.

May I ask why you’re looking for logs using api? Compared to executing a search on Web UI.
Could you describe your environment in which Graylog reside in?

3

The idea is to get the latest log entry for a service/pod, just to compare it with the log entries from the files and make sure the logs actually get delivered & measure the lag.
This search is preformed by a separate service/daemon, which then propagates metrics on each service/pod known to it.
I can limit the time range, but of course it would be better to use no limit at all.

4

I didnt think of that, nice. :slight_smile:

Need to ask a couple question.
What version of Graylog do you have?
What type of installment of Graylog?

5

Graylog version: graylog:4.0.6-1
It’s a self-hosted installment.

6

Hello,
Ive been testing this out on my lab server. I created a new server with GL 4.0.11, Es 7.10.2 and MongoDb 4.4.

I assigned 2GB to Heap and configured Es with 4GB (8 GB total on the node). The total amount of CPU’s using for this test was 10 cores. In my GL config file I set process_buffer to 5 Output Buffer to 3 and Input to 1. I left on core for the system. I did see the cpu go up for a bit when I execute the search but went down quick. Not sure if this will help you out but I’m unable to get your same results.
Sorry I cant be more help

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.