How do I determine the size of a field I add during Pipeline processing? I know there is the ES plugin that creates a new _size field that shows entire message size, but I don’t need to know how big my messages are indefinitely nor do I want to use up more Licensing volume with a meta field. I just need to calculate once during development how many bytes my new field and value use.
Here’s my Pipeline Rule:
rule “example”
when
// List of conditionals:
true
then
// Create new field, set value to Boolean True/False:
set_field(“test”, true);
end
Note that the value for the new “test” field is a Boolean not a String. So a value of True would be the same size as a value of False. If these were Strings, “False” would be 1 byte larger than “True” since it has 1 more ASCII character.
So would my new "test"=true field/value pair amount to 8 bytes (includes quotes and equal sign), or would it take up 11 bytes (translating Boolean True to String “True”)?
My first thought would be 8 bytes because True is Boolean. However when that log is archived, what happens to the ‘True’ value? I don’t see how it can be retained as a Boolean since it is exported to a G-zipped tarball and takes up 4 spaces when printed to the Linux terminal.
So in regards to Graylog Licensing volume, how many bytes would"test"=trueuse?
Oh sorry, for some reason none of my search results found that page.
So I see that Booleans are 4 Bytes, great. But I’m still confused about how strings are calculated. The docs say:
Each of the counters follows these rules:
count the length of the field name.
If the content of the field is a string, the length of the string is counted not the bytes of that string
So what is the relationship between string count and bytes? I thought every string character is always 1 byte, so what does it mean when it says it counts the length but not the byte size of the string?
Also, I see the counters in Graylog, but I don’t see how I can get a counter for a single particular message. As far as I can tell, the counters are for all logs overall. But thanks for this, it could definitely be useful.
So what is the relationship between string count and bytes? I thought every string character is always 1 byte, so what does it mean when it says it counts the length but not the byte size of the string?
you can have characters with more than one byte … so we count the actually length and not the size.
Also, I see the counters in Graylog, but I don’t see how I can get a counter for a single particular message. As far as I can tell, the counters are for all logs overall. But thanks for this, it could definitely be useful.
What counter did you mean? Graylog has multiple counters but it does not have a counter for a single message. Like how long does processing run for this message … because how should that scale in 60k/s environments?
