Pipeline help - Route to different index

Graylog Central (peer support)
,
1

So trying to route my local syslog messages routed to a different input so that I can set a different retension schedule and the pipeline that I have setup is acting unexpectedly:

I have indexes “Main” and “Local” and streams “All messages” (index set Main) and “Local Messages” (index set Local)

My Local input “appliance-syslog-udp” is receiving messages from my local server. When I attempt to use a pipeline rule to route these messages to the “Local Messages” stream and then remove them from the “All messages” stream it is putting messages that came from my global input “WinLogs-gelf” into the “Local Messages” stream.

My rule I have defined are as follows if anyone can see where I went wrong or suggest an alternative I would be very appreciative:

rule "function move main to local"
when
    from_input(id:"\<InputID\>")
then
    // the following date format assumes there's no time zone in the string
 route_to_stream(id:"\<LocalStreamID\>");
 remove_from_stream(id:"\<MainStreamID\>");
end
2

It’s not possible to remove a message from the default stream using the remove_from_stream() function in Graylog 2.4.x.

This will be possible in Graylog 3.0.0:

3

So I am obviously coming at this from a wrong angle.

My local input “appliance-syslog-udp” utilizing port 514 is sending messages into my main index. I am trying to get all of those messages into a different index so that I can define a different retention policy. I thought I could use route_to_stream (into a stream with my different index set) and remove_from_stream (removing from the All messages stream).

My problem seems to be with the pipeline I have created. I now have messages in the needed index but they aren’t the messages that are coming into the “appliance-syslog-udp” input. How would I go about figuring out what the pipeline is triggering off of?

Thanks for your help!

4

Can anyone answer this question?

How can I (using pipelines or some other means) set different retension for some messages?

5

You can use the pipeline simulator to find out how your pipelines modify the messages.

7

Ok, using the pipeline simulator I took a message that it looks like the pipeline triggered but getting the same "not satisfied Rule message

“Evaluation not satisfied Rule ‘function move main to local’ (5b1029c2cb93e5170abc6d81) in Pipeline ‘Main to Local Index’ (5b100216cb93e5170abc427b)”

Here is my message that I got from my logs that it triggered (but not in simulator):

{
  "version": "1.1",
  "AccountName":"SYSTEM",
  "AccountType":"User",
  "Channel":"Application",
  "Domain":"NT AUTHORITY",
  "EventID":"1035",
  "EventType":"INFO",
  "Keywords":"36028797018963970",
  "Opcode":"Info",
  "ProcessID":"0",
  "RecordNumber":"10787815",
  "Severity":"INFO",
  "SeverityValue":"2",
  "SourceModuleName":"in",
  "SourceModuleType":"im_msvistalog",
  "SourceName":"MsiInstaller",
  "Task":"0",
  "ThreadID":"0",
  "UserID":"S-1-5-18",
  "source":"\<SourceHostname>\",
  "full_message":"Windows Installer reconfigured the product. Product Name: Java Auto Updater. Product Version: 2.8.141.15. Product Language: 1033. Manufacturer: Oracle Corporation. Reconfiguration success or error status: 0.",
  "timestamp": 1385053862.3072,
  "level": 6,
  "message":"Windows Installer reconfigured the product. Product Name: Java A"
}

Here again is my pipeline rule:

rule "function move main to local"
when
    from_input(id:"3A58779f40b85fe8065554aa6f")
then
    // the following date format assumes there's no time zone in the string
 route_to_stream(id:"5b100261cb93e5170abc42cd");
 remove_from_stream(id:"000000000000000000000001");
end

Where am I going wrong?

8

Try using all lower case letters.

9

So looking at differences between Pipelines and Stream rules. Is it true that Stream rules are as the messages are coming and Pipeline rules are after the messages are already ingested?

10

That depends on the message processor order in your Graylog cluster (see System/Configurations).

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.