GELF TCP Input Data to specific index


What I aim is that what loads on TCP Input “Oracle Audit TCP Input” goes to index “Oracle Unified Audit”.

last image pipeline rule

Ok, for future reference those screen shots are really, really hard to read. I had to copy them and use another application to read it.

For testing purposes lets use a unique FIELD call DB_NAME as shown below.


Now create a this pipeline rule.

Rule "Route to stream"
when
    has_field("DB_NAME") AND contains(to_string($message.DB_NAME), "dblive01")
then
    route_to_stream(name: "Oracle Audit Stream", remove_from_default: true);
end

NOTE: If the stream name “Oracle Audit Stream” does not work you may need to us the Stream ID instead.

route_to_stream(id:"the_ID _of_the_stream");

To get the ID click on the stream you want and the ID would be located in your URL as shown below

image

NEXT
make sure your configuration is set with the pipeline processor after message filter chain
Something like this.

To find your Message Processors Configuration Navigate to System/Configurations.
I haven’t used from_input in my rules yet but I wanted to see if you pipeline is able to route to a stream first, I think this is the easiest way testing your pipeline out . Then we can go from there.

EDIT: You may need to wait a few minutes to see results, this depends on your environment.

I tried with both stream name and id. in both cases the message went to All Messages.

|3|Pipeline Processor|active|
|4|Message Filter Chain|active|

Should be this way

|3|Message Filter Chain|active|
|4|Pipeline Processor|active|

They are in default order, as per Graylog install. I have not changed anything.

You need this order.


Click the “Update” button and adjust it.

thank you @gsmith

but before I do that, is there any change oi this order on Graylog versions newer then mine - 3.3.2 ?

I would not like my future content pack to be in need of changing a default setting of the system.

No its default, and if you look in the forum other community members had the same issues, this order if preferred when using a pipeline. So long as your Message Filter Chain is in front of your Pipeline Processor you good. In your case something like this.

I am sorry but this time I cannot agree. If it has to be changed it means it is not the default.

I agree its not default setting but what you posted is default settings. As I stated before.

Well , if you don’t want to go that route you can configure your stream to something like this.
Make stream.

Configure rule on stream

Click save see if that works for you.

" I agree its not default setting but what you posted is default settings"

Sorry again, but I am not getting the above.
Looks like you have a later version of Graylog. Do you have by default a different order then mine (3.3.2), or would you have to do the same change to achieve what I need?

I had to re-configure my Messages Processor order for my pipeline to work as needed.

EDIT: please take a look here for a HowTo

https://docs.graylog.org/docs/usage

Also another community member had a familiar issue
.

On my lab, I have a list of pipeline rules attached to the “All Messages” Stream that direct messages to different streams, like a dealer giving out cards. Each of those rules also dictate whether the message also remains in the All Messages Stream or is removed when the message is directed to a different stream. The ordering of the rules can be quickly edited if I find a need. I can look at one screen and see which stream(s) messages from any given Input are going to.

By comparison, using Stream rules for this purpose would mean these rules are decentralised and require lots of poking about different Streams pages to figure out what is going on. Stream rules are something of a legacy feature in my mind, as Pipeline rules are just easier to use for this purpose and don’t require management of extractor order etc.

I agree about stream rules but the moral of the story is and correct me if I’m wrong. @altink has a device sending logs which ends up in all messages. @altink created a new index and attached a stream to it. @altink wants to route only a specific device/s logs to the new stream/index.

As shown above I wrote out the pipeline to route the messages for him but unfortunately it did not work. Searching through the forum the one problem that does occur most and is also shown in the documentation is the position where the Pipeline Processor is locate within the Message Processors Configuration. I also suggested to reconfigure it but unfortunately my demonstration maybe interrupted incorrectly. @altink stated,

“I would not like my future content pack to be in need of changing a default setting of the system”

.If that’s the case, I suggested using stream /w rules instead. I’m not sure how to solve this issue or maybe I’m misunderstanding the situation.

For your solution to work one needs to change a default setting in Graylog server. Be this the All Messages stream or the config ordering.
Meanwhile I intend to build a Content Pack - which other people (users) will use. I am not trying to see (and fix my issue) from the perspective of a system admin on a specific environment. I see this from a perspective of a Developer towards its users. Very few users would accept to change the defaults of their systems to accommodate a new solution.

Thus I asked (topic 21814) about the application_name field. If that is OK, I will use that field to isolate my app logs from anything other. And may be I will abandon the idea of using a separate index. That is the way I use in Splunk to do the same - app isolation. Thus I will not use even pipelines and pipeline rules.
Even if I use my index - I will use stream rules based on application_name=“my_app” to send any log with that field to my index. What remains is that I cannot differentiate between different inputs, all of them will send the event to my index if the application_name=“my_app”. After all, the later is no problem. If it has the right app name let it come to my index.

regards
Altin

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.