Logs not being written to the index

[joseraeiro]

1. Describe your incident:

I’ve configured an Input for Wazuh and it’s working fine, when I click on Show Received Messages I do get all the log entries coming from Wazuh.

I then created two indices, because this a multitenant architecture.

Which have, a stream each:

I then went to the Stream configuration and tested to see if the rule was being fired. It was:

But, alas, no log files are being written in the Index.

2. Describe your environment:

• OS Information:
  Linux Ubuntu Server 22.04

• Package Version:

Graylog v4.3.11+b48cf45

3. What steps have you already taken to try and solve the problem?

I have created and recreated the streams and indices, to no avail.

4. How can the community help?

Help me troubleshoot the problem.

Thanks in advance, guys!

[chris.black-gl (Chris Black)]

How are you routing those messages to the index? You must either enable index routing from the stream properties or do it via pipeline rule.

Remember to remove it from the All-Messages stream, or you’ll end up storing the same messages twice.

[joseraeiro]

@chris.black-gl

How are you routing those messages to the index?

All I did to route these messages to the index was referred to in my previous post.

Here, I have the index that’s accumulating Querys (I see the number of ops going up) but I don’t receive any ops on the Index field:

How can I enable index routing from the stream properties or do it via pipeline rule? I can’t seem to find this information anywhere!

Could you please give me some pointers and/or redirect me to a tutorial on how to achieve this?

[gsmith]

Hey

From the error I see above perhaps is causing this. Specially with timestamps. I would check Elasicsearch/Opensearch logs. See if you can find more information.

[joseraeiro]

@gsmith

From the error I see above perhaps is causing this

Perhaps what?

Thanks, I’ll check!

[joseraeiro]

I check the OpenSearch logs but didn’t find anything that caught my eye, except for these entries:

[2023-01-31T10:06:44,764][INFO ][o.o.c.m.MetadataIndexTemplateService] [node-1] adding template [winprovit-template] for index patterns [winprovit_*]
[2023-01-31T10:06:44,992][INFO ][o.o.c.m.MetadataCreateIndexService] [node-1] [winprovit_0] creating index, cause [api], templates [winprovit-template], shards [4]/[0]
[2023-01-31T10:06:46,401][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[winprovit_0][2], [winprovit_0][3], [winprovit_0][0]]]).
[2023-01-31T17:35:37,502][INFO ][o.o.c.m.MetadataIndexTemplateService] [node-1] adding template [winprovit-template] for index patterns [winprovit_*]
[2023-01-31T17:35:37,875][INFO ][o.o.c.m.MetadataCreateIndexService] [node-1] [winprovit_1] creating index, cause [api], templates [winprovit-template], shards [4]/[0]
[2023-01-31T17:35:38,070][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[winprovit_1][2], [winprovit_1][3], [winprovit_1][0]]]).
[2023-02-01T10:01:08,690][WARN ][r.suppressed             ] [node-1] path: /winprovit_1/_mapping, params: {index=winprovit_1}

I tried to add another Index and another stream, this time to match a rule for searching on the logs for a static field that I added to the input. And this is working, it is writing the logs to the index defined.

Also, when clicking the Name of the Stream I get no log entries in the defective streams:

Although I get them on the new stream:

I’ve also noticed that changing the stream rule to another one (e.g. this one:

stops the stream being written to the index.

I’m only able to stream the logs if I define the rule to match the static field added to the stream. If I try to use any other rule the logs aren’t streamed!

How can I troubleshoot this?

[joseraeiro]

I found the bug!

I was using a . as the Key separator on the JSON decoder and, in spite the fact that Graylog is writing the logs using _ as the Key separator, it somehow attributed the . as the Key Separator and hence wasn’t recognizing the Field value from the Edit Stream Rule dialogue which was defined as using the _ as the Key separator (agent_name).

Now it is writing the logs to the index.

Thanks a lot for your kind help @gsmith and @chris.black-gl!

[gsmith]

Awesome,glad you reolved it

[chris.black-gl (Chris Black)]

Glad you resolved the issue, but I thought I would answer your question about index routing. It’s in the Stream>Edit configuration box. You have to first create the target index on the System/Indices page. Then you go to Streams>Stream Name> Edit Stream. The dialog box looks like this:

You just choose the index name you want it to route into, then check the remove from All Messages box (or you’ll store two copies of each message that comes to that stream).