I was having issues with the default settings for the Syslog UDP input so I switched to using the Raw/Plaintext UDP input and will set the fields I want using pipelines/rules. One issue I have is that I can’t seem to find a way to set the facility and log level fields. These values are found in the header portion of the packet so ideally I should be able to pull the values from there (RFC5424). The log level is in the message so technically I could get it from there, but that doesn’t solve the issue of getting the facility as well. Is there a variable like $header available in pipeline rules similar to $message where the facility and log level values are stored? As far as I can tell, the only variable available in pipeline rules is $message and a few sub indices like gl2_remote_ip, source, etc.
Description of steps you’ve taken to attempt to solve the issue
I’ve looked through the pipeline documentation and the community forums but have not been able to find a solution to this issue.
You can easily extract facility and severity from PRI (priority) header from syslog still if you use Raw input. But your message should contain <PRI> in the beggining of your message. You can you either regex or GROK to extract it and then use pipeline function expand_syslog_priority to extract facility and severity.
Aha, that’s what I was looking for. So the value between the “<” and “>” is the priority. I tried out your rule and it works. However, I tried it with “expand_syslog_priority_as_string” and it doesn’t seem to work that way. When I hit apply, immediately the messages stop showing up in the stream. It works in the simulator though.
rule "priority parsing"
when
true
then
set_fields(grok(pattern: "<%{NONNEGINT:syslog_pri:int}>", value: to_string($message.message), only_named_captures: true));
let priority_string = expand_syslog_priority_as_string(to_long($message.syslog_pri));
set_fields({facility: priority_string.facility, log_level: priority_string.level});
end
gl2_processing_error
For rule 'priority parsing': In call to function 'expand_syslog_priority_as_string' at 7:24 an exception was thrown: For input string: "null"
Why would it be null in the “expand_syslog_priority_as_string” function, but not “expand_syslog_priority” function?
It also seems that I can’t set “facility” as anything except an integer, but I know that the Syslog UDP input uses the named string forms of the facility values such as local0 or local7.
Okay, the issue seems to be with setting the “facility” value. If I change it to something like “facility_string” the set_field command works. It just doesn’t let me set the “facility” value as a string, but I can set it as an integer. What I want the facility field to be though is the named value like local7, etc. just like the Syslog UDP input does. It looks like someone else may have come across this same issue, but it doesn’t appear to have been resolved.
I not a bug/issue or problem, it is a feature. ElasticSearch uses dynamic mapping to determine type of data, so if you first use one type of data (e.g. numeric), that type if remembered for this field. So another time you can’t insert another type of data (e.g. string) to same field. If you want to change type of data of the field, you have to use custom index mapping and then rotate index. Or save data to new field with different name, as you’ve already done.
Okay, so because Elasticsearch first received an integer for the “facility” field, the dynamic mapping set the field to “type” : “long” and as a result, only integers can be stored there now.
To correct this, I would have to force the “type” to “keyword” or “text” with a custom index mapping and then rotate the index so the new mapping will take effect. Do the dynamic mappings reset every time the index rotates?
The dynamic mappings do seem to reset when the index rotates. I set my pipeline rule back to saving the facility as the string value. The stream stopped showing any logs. Then I rotated the index and logs started pouring in again. I checked the dynamic mappings and now the facility is “type” : “keyword”. I’m sure you were referring to the custom mappings not resetting on rotation, but it looks like the dynamic ones do.
