Determine log source of caused error message ("Limit of total fields [1000] in index [graylog_000] has been exceeded")

[robertbln (Robert)]

Hello there,

I can’t figure out which exact log sources is causing the following error message:

java.lang.IllegalArgumentException: Limit of total fields [1000] in index [graylog_000] has been exceeded"

Of course I can increase the total fields size to a higher number, but this slows down Graylog (regarding to the instructions on What to Do When You Have 1000+ Fields? | Graylog)

I already use a separate stream with a separate index for specific (big) log sources I’m aware of (that works fine).

But such error messages still occur sometimes, which come from log sources I can’t recognize.

Does somebody know an apprach how to find it out?

Thanks in advance!

[jan (Jan Doberstein)]

he @robertbln

you should rotate the first index and check if you still run into the problem on the new index as the field counter start new for that.

Without any knowledge what you ingest and how it is only guessing what causes the problems.

[robertbln (Robert)]

Thanks for you answer. I have rotated the index, but in the new index I get always the same error messages.

[cawfehman]

without knowing much about your environment, I would say you either need to rotate indices quicker (not a great solution) or use more indices to handle the inputs.

upping the total field size is a bad idea not just from a performance perspective, but if you keep adding sources, you could eventually hit that cap too.

The last suggestion would be to see if there is any normalization you can do. Do you have a schema? do you have/need src_ip, source_ip, SrcIP, ip_source, etc… or can you normalize it to only use source_ip?

[robertbln (Robert)]

Thanks very much. I have split up everything in significant more indices, which has worked out!