I have an issue where messages in one stream disappear when I have multiple stages on a pipeline attached to it. With disappear i mean that they are just gone, they’re not in the stream nor in the actual input. I don’t really understand why they disappear from the input since I thought streams came after input and that the order was input => pipeline => stream.
The pipeline is rather simple. If I ONLY have stage 0 it works.
Stage 0 looks up IP on “source” and translate to hostname with the help of lookup table. Rule matching is “None or more rules on this stage match”
rule "internal ip to host"
when
has_field("source") && contains(to_string($message.source), "192.168.")
then
let lookupHostname = lookup_value("ip_to_hostname_lookup", $message.source, $message.source);
set_field("source", lookupHostname);
end
If I add another stage here like the simple snippet below the messages for this input mysteriously disappears. Can’t wrap my head around to why. I cannot see any errors anywhere but not sure where to look to be honest.
rule "test stage 1 dissapearance"
when
has_field("source")
then
// set fields
set_field("Location","hello world");
end
I wrote initially that it affects some streams and not all. If the message comes in with a proper hostname that’s NOT affected by rule 0 it looks like it’s not lost in the void.
Can someone point me in the direction to what I’m missing here. Probably something obvious but I’m still new to graylog and have somewhat of a hard time to get a good overview of it.
Quick clarification. When I write that it disappears from the input i mean that I do no longer see incoming messages when i filter on the gl2_source_input for that particular input. If I remove stage 1 I see them again.
Hmm. Seems like they disappear when stage 1 evaluates to true. I’m clueless to why.
This makes the messages vanish.
rule "test stage 1 dissapearance"
when
is_number(1)
then
// set fields
set_field("Location","hello world");
end
But this does NOT make them disappear
rule "test stage 1 dissapearance"
when
is_number("dfsdf")
then
// set fields
set_field("Location","hello world");
end
Maybe I don’t understand the set_field function. . Adding text in prefix and suffix makes the message appear. But empty prefix and suffix makes the message disappear.
So this works.
rule "test stage 1 dissapearance"
when
is_number(1)
then
// set fields
set_field("Location","hello world","pre","suf");
end
But leaving prefix or suffix out or just writing them as “” does not. I don’t want neither of them so not sure how to go about this. =/
rule "test stage 1 dissapearance"
when
is_number(1)
then
// set fields
set_field("Location","hello world","","");
end
In simulation it works fine with set_field(“Location”,“hello world”) and all looks fine and dandy. But in “live environment” my messages just disappears and are nowhere to be found. =(
You can use the debug() function to watch what is going on in each rule. If you do a search for the message without defining the index are you able to find it… sometimes it will get dropped in “All Messages” but it may also get dumped into “Processing and Index failures” and then would likely have a field in it describing the issue it had.
Not sure debug() is helpful here. Reading the description for it;
Print any passed value as string in the graylog-server log. Note that this will only appear in the log of the graylog-server node that is processing the message you are trying to debug.
So I can only use that to check the value of a string/variable. according to the description it will not give me a detailed debug of that message/rule. In my case it’s not a value I need to trace, I need to trace the message itself.
I have all ‘failure processing’ enabled but i don’t have anything in that stream. I also can’t see the message in ‘all messages’. So I’m not really sure what’s going on here, smells like a bug to me but graylog is rather complex and I’m new so might be me missing something obvious.
I spooke too soon. I added a debug(“hello world”) before and after the set_field() and got this in the log.
2022-01-10T18:30:26.662+01:00 INFO [Function] PIPELINE DEBUG: hello world before
2022-01-10T18:30:26.663+01:00 INFO [Function] PIPELINE DEBUG: hello world after
2022-01-10T18:30:26.993+01:00 ERROR [MessagesAdapterES7] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [env-apc_0], type [_doc], id [ffc80151-723a-11ec-b87b-525400c96922], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=object mapping for [Location] tried to parse field [Location] as object, but found a concrete value]]]
[1]: index [graylog_9], type [_doc], id [ffc80151-723a-11ec-b87b-525400c96922], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=object mapping for [Location] tried to parse field [Location] as object, but found a concrete value]]]
Using set_field with another field makes it work. So something with ‘Location’ as name that screws it up in this particular stage. I have had it to work and written to Location so some sort of collision with something old?
I can also see that the original location from hostname rule have a bunch of errors logged. Not sure where I see those errors of how I clear them though?
that original rule ‘location from hostname’ looks like this. But as I’ve written above, it’s not even in the pipeline now. I only have that simple rule with ‘hello world’ i wrote in previous post.
rule "location from hostname"
when
has_field("source") && contains(to_string($message.source), "-") && NOT contains(to_string($message.source), "DDN")
then
// regex to get short location up until "-" in hostname, i.e. m5, fh, jg
let extractedLocation = regex("^(.*?)-",to_string($message.source));
// look up in table
let location_multi = lookup("hostname-locations-lookup", extractedLocation["0"]);
// csv does not support multivalues properly. Split second value on delimiter
let location_split = split(";",to_string(location_multi.value));
// set fields
set_field("Location",to_string(location_split[0]));
set_field("Street",to_string(location_split[1]));
set_field("City",to_string(location_split[2]));
end
Well… so many questions! I have no idea what the actual issue was other then some sort for collision with an existing object in db?
Rewrote that rule with new field-names and it SEEMS to work now. Also noticed that you can messages dissapear without errors in log if you have something like “,” in your set_field name. I find it somewhat scary with silent errors like this and no clear way to see it in graylog. Makes it hard to trust the system even if this was some sort of unknown error by me…
rule "location from hostname"
when
has_field("source") && contains(to_string($message.source), "-") && NOT contains(to_string($message.source), "DDN")
then
// regex to get short location up until "-" in hostname, i.e. m5, fh, jg
let extractedLocation = regex("^(.*?)-",to_string($message.source));
// look up in table
let location_multi = lookup("hostname-locations-lookup", extractedLocation["0"]);
// csv does not support multivalues properly. Split second value on delimiter
let location_split = split(";",to_string(location_multi.value));
// set fields
set_field("location_site",to_string(location_split[0]));
set_field("location_street",to_string(location_split[1]));
set_field("location_city",to_string(location_split[2]));
end
Elasticsearch holds on to the field type that it guesses an object is when each successive index is created. If you rotate your index it will find the new type… You could do a custom mapping to force the type you want and even retroactively change old data… though it is a lot of work. Unless you need to go back, the best way is to do what you did, creating new fields that are specific to the data you want. Did you check “Processing and Index failures” stream to see if the issues were getting shunted there?
. Adding text in prefix and suffix makes the message appear. But empty prefix and suffix makes the message disappear.
So I’m not really sure what’s going on here, smells like a bug to me but graylog is rather complex and I’m new so might be me missing something obvious.
I have no idea what the actual issue was other then some sort for collision with an existing object in db?