Lookup result always return null for single value jsonpath

Graylog Central (peer support)
,
1

Hi All,

I am using HTTP JSONPath as Data Adapter Type in lookup tables to get single value lookup but it’s not working as expected.

We have created API using node.js and below is the url and its response,

https://localhost:9000/dsl/api/elasticsearch/getNodeById?key=${key}

Response

[
    {
        "_index": "test-node",
        "_type": "nodes",
        "_id": "18216d80-b9d476-7a0e9233716b",
        "_score": 7.926042,
        "_source": {
            "parent_id": "RD00155DF24AE6",
            "id": "RD00155DF24AE6::IIS_Access",
            "uuid": "18216d80-b9d76-7a0e9233716b",
            "timestamp": "2019-08-09T20:05:17.973079Z"
        }
    },
    {
        "_index": "test-node",
        "_type": "nodes",
        "_id": "99aa2220-76-7a0e9233716b",
        "_score": 4.9414196,
        "_source": {
            "parent_id": "RD00155D6AF6D1"
            "id": "RD00155D6AF6D1::IIS_Access",
            "uuid": "99aa2220-d76-7a0e9233716b",
            "timestamp": "2019-08-09T20:05:18.594186Z"
        }
    }
]

I am trying to test the lookup in graylog but the single & multi value always return “null” and below is the screenshot reference,

image
image.png1336×506 45.9 KB

Please correct me if am doing anything wrong and help me to resolve the issue.

Thanks,
Ganeshbabu R

2

you might need to urlendcode your special character (the ::) in the Key

3

Hi @jan

We changed the data format and “id” key will have values like below, and since the API is going to hit elastic and we are getting the below response,

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 6.3664703,
        "hits": [
            {
                "_index": "test-node",
                "_type": "nodes",
                "_id": "a47d5e40-bd00-42d9c4e62d63",
                "_score": 6.3664703,
                "_source": {
                    "uuid": "a47d5e40-bd00-42d9c4e62d63",
                    "parent_id": "RD00155DF24AE6",
                    "id": "RD00155DF24AE6",
                    "timestamp": "2019-08-26T10:51:01.409543Z"
                }
            }
        ]
    }
}

then now i tried test lookup in graylog by giving single value JSON path as $.hits.hits[0]._source.id" and data is showing now…

image
image984×460 34.4 KB

But now i am facing another problem,

I have a pipeline (testbeat) and test stream is connected to that and it has two rules to it.

Rule1:-

rule "testbeat"
when
    to_string($message.set_name) == "process"
then
    let msg = create_message("", "");
    let new_date = to_string($message.timestamp);
    let id = to_string($message.host_name);
    let parent_id = id;
    let test = "testbeat";
    set_field("@timestamp", new_date, "", "", msg);
    set_field("id", id, "", "", msg);
    set_field("parent_id", parent_id, "", "", msg);
    set_field("test",test, "", "", msg);
    route_to_stream("testbeat", "", msg);
end

Rule2:-

rule "nodeid_lookup"
when has_field("test")
then
  let nodeid = to_string($message.id);
  let id = lookup_value("nodeid_lookup", nodeid);
  set_field("nodeID", id);
end

Once the messages are processed I can’t see the new field nodeID in my message and its not adding it and please guide me if i am doing anything wrong in the config of lookup and correct me if my understanding is wrong.

Thanks,
Ganeshbabu R

4

Almost every time I create a new lookup (or similar) I add a debug field with the lookup value as well just to ensure the value I want to lookup is what it should be:

set_field("DEBUG_lookup_value",nodeid);

Then at least you know which direction to head for troubleshooting.

If it is there - you can look at your lookup setup, and if its not then your pipeline logic is funky.

5

Yes I did the same and the above I am trying to set field nodeID to my message using lookup_value function but didn’t worked. I changed the rule with the below,

rule “nodeid_lookup”
when has_field(“test”)
then
let nodeid = to_string($message.id);
let id = lookup(“nodeid_lookup”, nodeid);
set_field(“nodeID”, id);
end

Once the messages are processed now I can see the new field nodeID in my message but not sure why its not working when I used lookup_value function. I need to debug on this lookup setup.

Thanks,
Ganeshbabu R

6

Next step is to create a default value in your lookup table.

This means your rule logic is sound. If you set up an ‘unknown’ default single and an ‘unknown-multi’ value then you can confirm that you’re actually connecting to the lookup table.

IF you look at the example in the very bottom right at the url:
https://YOUR_GRAYLOG_SERVER/system/lookuptables/data_adapter/NodeId_Adapter/edit

You can see an example pipeline rule ( as we can’t see all your names from your screenshot). I imagine if you see the default values I mentioned, then its simple a naming-context issue in your pipeline rule.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.