Journal duplicate messages?

Graylog Central (peer support)
1

Hi,
I’m using Graylog 4.3.7. There is a strange behavior I saw during log data ingestion.
Using GELF input I sent data from a log file with about 1,800,000 rows.
Graylog installation with active journal.

The strangeness is that on graylog are present about 2,600,000 messages
What I can see is that the journal was full (more that 100%)

After that I test the same ingeston disabling journal.
Now the messages count is exactly the file row number (1,800,000).

So, is the journal the responsible for this strange behavior?
Can somebody help me to understand what happened?

Gianluca

2

I saw this one: Graylog journal getting full
Maybe it works for you, too.

1 Like
3

on what index are those data? And what is the data-retention? Do you rotate by number of logs or by time, and how many rotations do you do, before you delete?

4

Hi @ihe
here the info you asked.

image
image732×65 5.93 KB

I deleted all elastic indices

So the question is: why with the journal I had that biggest number of messages (2,800,000 vs 1,800,000 of the file)?

5

Hard to guess without more information, but it is the first time I have ever heard if that happening.

I understand that GELF is the format, but how did you send the messages? Any chance the messages were duplicated by the sending mechanism?

If you repeat what you did the first time, does it produce the same duplication?

6

Hi @joe.gross,

Thanx for your time. I did a bulk ingestion from file with about 1,800,000 rows.
With journal on, it become full and when I see the message count on the graylog elastic or using itessage count i see that number difference.

Removing the journal the number of file ow and messages is the same.

Adding the journal the number is different. The message count is not always the same but it is greater than the file rows.

I use logstash to ingest file rows
Thanks
Gianluca

7

Hmm. Not doubled then? How much over 1.8M is it? Does it vary?

Exactly how/where are you getting your counts? Please be specific and detailed.

How are you removing the journal? Again, please be specific and detailed.

Finally, Is the journal reaching 100% when you do the bulk import?

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.