Pipeline problem in search

Hello all,

I don’t understand why I can’t showing up pipeline in search tab, the pipeline test is ok and the order in system-configurations is:

|1|AWS Instance Name
|2|GeoIP Resolver
|3|Message Filter Chain
|4|Pipeline Processor

Pipeline is connected to streams “all messages”
any suggestions?

thanks

Hello,
My Message Processors Configuration looks like this.

Maybe give that a try. Unless you using your AWS lookup mine is disabled.
Hope that helps.

thanks but unfortunately doesn’t work.

I try the same pipeline for some syslog messages and it works but I need this for a “Raw/Plaintext AMQP” input, so it’s the graylog that reads the Rabbit queue and I think I need to do something different for this input.

Hello,
Can you share your INPUT, maybe a message and Pipeline configuration?

Thanks

this is an example message:
immagine

i need to create a field named event when match 0x10000 and set “createfile”

immagine

the input

immagine

If i tested with simulator all work

thanks

Hello,
If you just need to create a field from the message have you tried to use Extractors? Here is one example that I have.

Message

My extractor created for this INPUT.

The end result.

Hope that helps

yes i try, but i need to change the value in the field

if i have 0x10000 in message the value of the extractors will be “create file”

in extractors i didn’t find anything to do this

thanks

@nsecurity
Hello,
I tried to create an example for you. What I’m using is Graylog 4.0.
Created a new field using a Regular expression from what information you gave us as shown below,
First I used this site for testing my Regular expression → here.

Then I created the extrator.

Using a pipeline to convert the your_new_field to a different value.

rule "event"
when
    has_field("your_new_field") AND contains(to_string($message.your_new_field), "0x10000")
then
    set_field("your_new_filed","new_value");
end

Still fairly new at using pipelines and Regular expression. This worked for me to converting numbers into a name. I have seen other members/staff here work with pipelines better then i can.
Hope this helps

@gsmith
Hello,
I have updated graylog to 4.0 but still not working even with extractors.
If I use on message the funcion “add to query” this is the result:
image

image

the original message is in a strange format so, probably, did not match with extractors,pipelines ecc…
do you ever seen something like this?
thanks

@nsecurity

Agree, if the mesages looks like that, the extractor are not going to help.

Yes I have, when one of my INPUTs were incorrect. For an example: Using Syslog UDP INPUT for my frewalls I had to change my INPUT to Raw/Plaintext UDP.

You still using Raw/Plaintext AMQP? is so try using Raw/Plaintext UDP as an INPUT see if that helps. The order of what I would do is make sure messages are coming through correct. Then I would add the extractors to create unique fields. And last I would apply the pipeline. Just a thought…
Hope that helps

@gsmith

unfortunately I have to use Raw/Plaintext AMQP because the source support only rabbitmq for third party solutions and I can’t normalize logs.

thanks

How are you sending those messages to RabbitMQ? Do you happen to be sending Windows logs through RabbitMQ to Graylog? I ask because I ran into a similar issue (sans RabbitMQ) that seemed to indicate that it was actually an encoding issue on the Windows side. If it is in fact a Windows system, you’ll want to look at how that file is encoded. IANA Windows admin, but from what Google tells me, Windows uses UTF-16. If you’re using Filebeat, you should be able to set the encoding like so:

encoding: utf-16-bom

logs arrive from a EMC unity, rabbitmq is installed on a windows machine and talk with the EMC CEE service installed on the same machine.

I configure a Splunk endpoint that ricognized all the fileds of EMC logs without rabbitmq beacause CEE has a dedicated configuration for Splunk and these services communicate each other via HTTP.

I didn’t find anything like this in graylog so I try to do the same thing with rabbitmq.

thanks