Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic!
Hello!
Our graylog server receives user commands on a Linux box. We don’t have a lot of users, but the commands output the user as an ID i.e. 191460003 instead of a username. We thought using a lookup_table with a CSV which has the mapping of the UID’s to usernames would work great, but we’re having issues getting it to replace the UID’s in the stream.
The client sending the logs is on RHEL 7 and our server is running Graylog 4.2.5
3. What steps have you already taken to try and solve the problem?
We have setup the lookup table and running the test lookup correctly shows the username instead of the UID, so we think this part is okay. Am having some issues understanding the pipeline portion. My understanding is that we create a pipeline rule and then attach it to our stream.
Ultimately we’d want the rule to replace any UID that matches in the CSV and replace it with the username but was trying with a single UID to simplify it.
rule "UID to Username"
when
has_field("message") AND contains(to_string($message.uid),"1914600003")
then
//Change UID to Username
let update_source = lookup_value("lookup_username",$message.uid);
set_field("message",update_source);
end
Was using this as reference as thought it closely resembled what we are trying to do:
I have done something similar to what you want to do, except I have not used lookup_table.
What I can do is show my configurations and perhaps it will help.
Graylog uses UUID in rest.accesslog for user names, so what I had to do is get that UUID then create a field for it, then using a pipeline change the UUID into a human readable name.
rule "Graylog Web Access Greg"
when
has_field("graylog_gui") AND contains(to_string($message.graylog_gui), "5e224e7683d72eff75055199")
then
set_field("graylog_gui","greg.smith");
end
The rule is close but you are referencing “message” which is the entire message…
Here is how I would set the rule up - all the other things you said make sense
rule "UID to Username"
when
has_field("uid")
then
//Change UID to Username
let update_source = lookup_value("lookup_username",to_string($message.uid));
// note: I think you can set the default value of a table if the lookup fails to USER_NOT_FOUND
// set the field username to be the table results for uid
set_field("username",update_source);
end
Or you if you use the @gsmith extractor method, you could spoof an if statement in the rule like this so that you would only have to change one rule when names or uid’s change:
rule "Graylog Web Access User Convert"
when
has_field("graylog_gui")
then
// all these will fail to replace except the one that has the correct data...or note that user was not found
let new_field = "USER_NOT_FOUND";
let new_field = replace(to_string($message.graylog_gui), "5e224e7683d72eff75055199", "greg.smith");
let new_field = replace(to_string($message.graylog_gui), "100100110010011001001", "tad.sherrill");
let new_field = replace(to_string($message.graylog_gui), "987324e32874ff32892b2829", "steven.o");
set_field("graylog_gui",new_field);
end
I also will try this suggestion as well. I think the issue is that “uid” isn’t a field in our log. The UID is just a string of numbers taken from the body of the message of the log file. I was hoping to find a way to have the pipeline convert this string to a username when the value matches in a lookup table. Our lookup table has fields “id,username” but I’m guessing that the pipeline only knows about these fields when using something like lookup_value.
If you post an example of the full message, we can modify the rule using rule functions like regex() or similar to pull the UID value out of the message and then compare it in the table…
The following also works to convert the UID to username in a new field, “username” if unable to have it search the string for any matching values and do the conversion dynamically.
rule "UID to Username"
when
has_field("message")
then
//Change UID to Username
let update_source = lookup_value("lookup_username",to_string(1914600003),"USER_NOT_FOUND");
set_field("username",update_source);
end
Do you want to replace the UID int he message, add the username onto the message or have a separate field that contains the username?
I would write the rule so that if you need to add username/UID combinations you only need to change one rule or modify one table file - much easier to manage.
Ultimate goal would be to replace the UID with the username in the message itself. If unable, second best would be for it to be in a new field as we seem to sort of have it working now.
rule "Graylog Web Access User Convert"
when
contains(to_string($message.message),"UID=")
then
// create a local to make it slightly less intensive to work with
let robin = to_string($message.message);
// EDIT: deleted bad part
let batman = replace(robin, "5e224e7683d72eff75055199", "greg.smith");
let batman = replace(robin, "100100110010011001001", "tad.sherrill");
let batman = replace(robin, "987324e32874ff32892b2829", "steven.o");
// replace the hexadecimal that follows UID= with USER_NOT_FOUND....
// the d+ means it will only be found and replaced if it's a series of digits that includes chars a thru f.
// the a-f part takes into account your UID may be base 16 hexidecimal like Gregs
// if you are sure it's just digits, you can just use "(?<=UID\\=)(?>\\d+)"
// might be dangerous if your username has only letters a-f
let batman = regex_replace("(?<=UID\\=)(?>[\\da-f]+)",robin,"USER_NOT_FOUND",false);
// replace the numbers that follows UID= with USER_NOT_FOUND
// let batman = regex_replace("(?<=UID\\=)(?>\\d+)",robin,"USER_NOT_FOUND",false);
set_field("message",batman);
end
This is definitely looking closer, but it seems that its not able to correctly identify the string to replace it.
I slightly modified the code to use a test UID. Also, as you had suggested in your comments changed the “a-f” to “d+” since all of the UID’s will be numbers only.
rule "Graylog Web Access User Convert"
when
contains(to_string($message.message),"UID=")
then
// create a local to make it slightly less intensive to work with
let robin = to_string($message.message);
// replace the number that follows UID= with USER_NOT_FOUND
let batman = regex_replace("(?<=UID\\=)(?>\\d+)",robin,"USER_NOT_FOUND",false);
let batman = replace(robin, "1914600003", "steveno");
let batman = replace(robin, "100100110010011001001", "tad.sherrill");
let batman = replace(robin, "987324e32874ff32892b2829", "steven.o");
// replace the numbers that follows UID= with USER_NOT_FOUND
let batman = regex_replace("(?<=UID\\=)(?>\\d+)",robin,"USER_NOT_FOUND",false);
set_field("message",batman);
end
It is replacing the UID with USER_NOT_FOUND directly inline on the message which is what we had hoped for, but now just need to understand why it doesn’t identify the UID and replace it with the matching username as expected.
Ah, I think we’re in business now! It seems we needed to set “robin” instead of “batman” in the second section.
rule "Graylog Web Access User Convert"
when
contains(to_string($message.message),"UID=")
then
// create a local to make it slightly less intensive to work with
let robin = to_string($message.message);
// replace the number that follows UID= with USER_NOT_FOUND
let batman = regex_replace("(?<=UID\\=)(?>\\d{10})",robin,"USER_NOT_FOUND",false);
let robin = replace(robin,"1914600003", "steveno");
// let batman = replace(robin, "1914600003", "steveno");
// let batman = replace(robin, "100100110010011001001", "tad.sherrill");
// let batman = replace(robin, "987324e32874ff32892b2829", "steven.o");
// replace the numbers that follows UID= with USER_NOT_FOUND
let batman = regex_replace("(?<=UID\\=)(?>\\d{10})",robin,"USER_NOT_FOUND",false);
set_field("message",batman);
end
I messed up - take out the first “user not found” that deletes the number you are looking for and screws the whole thing up! - I had deleted it in my test but messed up on the one I pasted in!!! so it should be:
rule "Graylog Web Access User Convert"
when
contains(to_string($message.message),"UID=")
then
// create a local to make it slightly less intensive to work with
let robin = to_string($message.message);
let batman = replace(robin, "1914600003", "steveno");
let batman = replace(robin, "100100110010011001001", "tad.sherrill");
let batman = replace(robin, "987324e32874ff32892b2829", "steven.o");
// changed the number of digits count to be required between 6 and 10 {6,10}
let batman = regex_replace("(?<=UID\\=)(?>\\d{6,10})",robin,"USER_NOT_FOUND",false);
set_field("message",batman);
end
Thanks tmacgbay - that makes sense, when I update the code it still reports USER_NOT_FOUND unless I change the line
let batman = replace(robin, “1914600003”, “steveno”);
to
let robin = replace(robin, “1914600003”, “steveno”);
Here’s the rule in its entirety
rule "Graylog Web Access User Convert"
when
contains(to_string($message.message),"UID=")
then
// create a local to make it slightly less intensive to work with
let robin = to_string($message.message);
let batman = replace(robin,"1914600003", "steveno");
// let batman = replace(robin, "100100110010011001001", "tad.sherrill");
// let batman = replace(robin, "987324e32874ff32892b2829", "steven.o");
// replace the numbers that follows UID= with USER_NOT_FOUND
let batman = regex_replace("(?<=UID\\=)(?>\\d{6,10})",robin,"USER_NOT_FOUND",false);
set_field("message",batman);
end
This solution does exactly what we needed. Your time and patience was appreciated. We’ve decided to forgo the “USER_NOT_FOUND” as it’s probably better to have it to show the UID anyway so that we can identify the user manually if we haven’t updated the table yet.
