# Pipeline help - Route to different index

**URL:** https://community.graylog.org/t/pipeline-help-route-to-different-index/5422
**Category:** Graylog Central (peer support)
**Tags:** route-to-streampl, pipeline-rules
**Created:** [May 31, 2018, 6:26pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422 "2018-05-31T18:26:26Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![oden08](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/oden08/32/16018_2.png) [@oden08](https://community.graylog.org/u/oden08)
#### Post date: [May 31, 2018, 6:26pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/1 "2018-05-31T18:26:26Z")

</div>

So trying to route my local syslog messages routed to a different input so that I can set a different retension schedule and the pipeline that I have setup is acting unexpectedly:

I have indexes “Main” and “Local” and streams “All messages” (index set Main) and “Local Messages” (index set Local)

My Local input “appliance-syslog-udp” is receiving messages from my local server. When I attempt to use a pipeline rule to route these messages to the “Local Messages” stream and then remove them from the “All messages” stream it is putting messages that came from my global input “WinLogs-gelf” into the “Local Messages” stream.

My rule I have defined are as follows if anyone can see where I went wrong or suggest an alternative I would be very appreciative:

```nohighlight
rule "function move main to local"
when
    from_input(id:"\<InputID\>")
then
    // the following date format assumes there's no time zone in the string
 route_to_stream(id:"\<LocalStreamID\>");
 remove_from_stream(id:"\<MainStreamID\>");
end

```

---

<div class="post-metadata">

### Author: ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [June 1, 2018, 7:38am UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/2 "2018-06-01T07:38:03Z")

</div>

It’s not possible to remove a message from the default stream using the `remove_from_stream()` function in Graylog 2.4.x.

This will be possible in Graylog 3.0.0:

> <https://github.com/Graylog2/graylog-plugin-pipeline-processor/pull/220>

---

<div class="post-metadata">

### Author: ![oden08](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/oden08/32/16018_2.png) [@oden08](https://community.graylog.org/u/oden08)
#### Post date: [June 1, 2018, 12:00pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/3 "2018-06-01T12:00:39Z")

</div>

So I am obviously coming at this from a wrong angle.

My local input “appliance-syslog-udp” utilizing port 514 is sending messages into my main index. I am trying to get all of those messages into a different index so that I can define a different retention policy. I thought I could use route\_to\_stream (into a stream with my different index set) and remove\_from\_stream (removing from the All messages stream).

My problem seems to be with the pipeline I have created. I now have messages in the needed index but they aren’t the messages that are coming into the “appliance-syslog-udp” input. How would I go about figuring out what the pipeline is triggering off of?

Thanks for your help!

---

<div class="post-metadata">

### Author: ![oden08](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/oden08/32/16018_2.png) [@oden08](https://community.graylog.org/u/oden08)
#### Post date: [June 4, 2018, 11:38am UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/4 "2018-06-04T11:38:23Z")

</div>

Can anyone answer this question?

How can I (using pipelines or some other means) set different retension for some messages?

---

<div class="post-metadata">

### Author: ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [June 4, 2018, 12:39pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/5 "2018-06-04T12:39:55Z")

</div>

> [@oden08](#):
>
> My problem seems to be with the pipeline I have created. I now have messages in the needed index but they aren’t the messages that are coming into the “appliance-syslog-udp” input. How would I go about figuring out what the pipeline is triggering off of?

You can use the pipeline simulator to find out how your pipelines modify the messages.

---

<div class="post-metadata">

### Author: ![oden08](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/oden08/32/16018_2.png) [@oden08](https://community.graylog.org/u/oden08)
#### Post date: [June 4, 2018, 5:30pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/7 "2018-06-04T17:30:13Z")

</div>

Ok, using the pipeline simulator I took a message that it looks like the pipeline triggered but getting the same "not satisfied Rule message

> “Evaluation not satisfied Rule ‘function move main to local’ (5b1029c2cb93e5170abc6d81) in Pipeline ‘Main to Local Index’ (5b100216cb93e5170abc427b)”

Here is my message that I got from my logs that it triggered (but not in simulator):

```auto
{
  "version": "1.1",
  "AccountName":"SYSTEM",
  "AccountType":"User",
  "Channel":"Application",
  "Domain":"NT AUTHORITY",
  "EventID":"1035",
  "EventType":"INFO",
  "Keywords":"36028797018963970",
  "Opcode":"Info",
  "ProcessID":"0",
  "RecordNumber":"10787815",
  "Severity":"INFO",
  "SeverityValue":"2",
  "SourceModuleName":"in",
  "SourceModuleType":"im_msvistalog",
  "SourceName":"MsiInstaller",
  "Task":"0",
  "ThreadID":"0",
  "UserID":"S-1-5-18",
  "source":"\<SourceHostname>\",
  "full_message":"Windows Installer reconfigured the product. Product Name: Java Auto Updater. Product Version: 2.8.141.15. Product Language: 1033. Manufacturer: Oracle Corporation. Reconfiguration success or error status: 0.",
  "timestamp": 1385053862.3072,
  "level": 6,
  "message":"Windows Installer reconfigured the product. Product Name: Java A"
}

```

Here again is my pipeline rule:

```auto
rule "function move main to local"
when
    from_input(id:"3A58779f40b85fe8065554aa6f")
then
    // the following date format assumes there's no time zone in the string
 route_to_stream(id:"5b100261cb93e5170abc42cd");
 remove_from_stream(id:"000000000000000000000001");
end

```

Where am I going wrong?

---

<div class="post-metadata">

### Author: ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [June 5, 2018, 6:54am UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/8 "2018-06-05T06:54:41Z")

</div>

> [@oden08](#):
>
> ```auto
> from_input(id:"3A58779f40b85fe8065554aa6f")
> 
> ```

Try using all lower case letters.

---

<div class="post-metadata">

### Author: ![oden08](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/oden08/32/16018_2.png) [@oden08](https://community.graylog.org/u/oden08)
#### Post date: [June 5, 2018, 5:04pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/9 "2018-06-05T17:04:55Z")

</div>

So looking at differences between Pipelines and Stream rules. Is it true that Stream rules are as the messages are coming and Pipeline rules are after the messages are already ingested?

---

<div class="post-metadata">

### Author: ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [June 5, 2018, 5:07pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/10 "2018-06-05T17:07:58Z")

</div>

> [@oden08](#):
>
> Is it true that Stream rules are as the messages are coming and Pipeline rules are after the messages are already ingested?

That depends on the message processor order in your Graylog cluster (see System/Configurations).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)
#### Post date: [June 19, 2018, 5:07pm UTC](https://community.graylog.org/t/pipeline-help-route-to-different-index/5422/11 "2018-06-19T17:07:59Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
