JSON Extractor is not working as intended

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
json extractor not working as intended where I have configured it as the following:

{
“extractors”: [
{
“title”: “WAZUH JSON EXTRACTOR”,
“extractor_type”: “json”,
“converters”: ,
“order”: 0,
“cursor_strategy”: “cut”,
“source_field”: “message”,
“target_field”: “”,
“extractor_config”: {
“list_separator”: ", ",
“kv_separator”: “:”,
“key_prefix”: “”,
“key_separator”: “.”,
“replace_key_whitespace”: false,
“key_whitespace_replacement”: “_”
},
“condition_type”: “none”,
“condition_value”: “”
}
],
“version”: “4.3.12”
}

The corresponding input is configured to receive messages from the wazuh manager. The problem is although the key separator is configured as “.” it appears with “_” in the corresponding stream.

Noting this is the only extractor configured for the input.
2. Describe your environment:

  • OS Information:
    Debian 11

  • Package Version:
    graylog 4.3.12

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
I have repeated the process of creating the input, extractor, and streams multiple times and tried to restart the input and stream but behavior didn’t change.

4. How can the community help?
Since I am new to graylog I need to find the reason behind this behavior and solve this issue. As I am using graylog to send alerts to the wazuh indexer this issue is causing errors in wazuh modules as it’s learned to parse alerts with “.” as key separator.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hey @Bilal-SG

Im not great at JSON extractors but have you tried to enable Flatten structures?

An old issue: GL replaces “.” with “_” before writing to ES, because historically ES did not allow “.” in field names.

There’s also an issue to change that for recent versions of ES, though I wouldn’t hold my breath:

1 Like

Hi Patrickmann,
Thanks for your reply. Are you saying this is still an open case? not solved yet?

Yes - the issues are still open.
Given how old they are, I am not sure when or if it will be addressed though. Feel free to add a comment to the issue, if this is a problem for you.

Hey gsmith,
Just tried to do it. But this is not what I wanted…

I just searched in ES guides and found this
Dots are reserved for grouping related fields into subdocuments
Use snake case (underscores) for combining words

So I am just wondering if this is the language ES understands, dots need to be the key separators and underscores are for combining words, and GL mainly needs to send logs to ES, why didn’t GL adapt to this? what is the point here, I am thinking?

I appreciate if you could answer my question…

It’s always difficult to prioritize issues, since we can’t get to everything; and this issue has not been a significant problem for users to date.
We should definitely improve the UI text in the extractor, since it is misleading. I will also resubmit the underscore issue to triage - it’s been lingering for a long time.

I might not have explained my point very well… it’s not the prohibition that I wanted but allowing it, instead. In my case, Underscores cause to me a big problem! Actually, I am viewing GL alerts from Wazuh dashboard. And since Wazuh dashboard depends on OpenSearch(OS), which understand dots as key separators and underscores as words combiners, it’s unable to lookup the fields correctly and output them!! So I am losing a dozen wazuh features because of this underscore!! From my perspective, all tools should speak the language of ES/OS in order to facilitate integrations between them and make it more useful and valuable!

I appreciate if you could add this in the case opened… Thanks!

I understand. There are 2 open issues now: one is to allow dots; the other is to inform users in the interim that dots are not allowed.

You could describe your scenario in issue 4583, so triage can understand the significance.

1 Like

Thanks, dear. I did it already.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.