# Failing JSON Extractor

**URL:** <https://community.graylog.org/t/failing-json-extractor/7944>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules, debuggingpl\
**Created:** [December 7, 2018, 12:24pm UTC](https://community.graylog.org/t/failing-json-extractor/7944 "2018-12-07T12:24:34Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![mulgurul](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mulgurul](https://community.graylog.org/u/mulgurul)\
**Post date:** [December 7, 2018, 12:24pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/1 "2018-12-07T12:24:34Z")

</div>

Hi

I’m quite new to Graylog configuration. I’m trying to set up a Filebeat input where there’s JSON single line events in the log files, like this:

{ “time”: “2018-12-07 12:56:12.2948”, “level”: “INFO”, “message”: “This is log event number 93 generated randomly in batch 374fa6d5-253a-4111-b274-f780a4255d47”, “CorrelationID”: “36835d49-c68e-4bb8-91e2-28954217e0a2”, “SessionID”: “6869282”, “Component”: “APP”, “ComponentVersion”: “1.5.0.0”, “Action”: “SEARCH”, “Method”: “SeachBook”, “MemberID”: “10820048”, “TimeSpent”: “” }

The sidecar is working, and when i submit logs I can see that they are received at greylog by looking at the in/out metric counter, and they appear in the input stream when searching, as long as I dont add an JSON extractor on the input.

I have understood that I need to add a JSON extractor on the message field. but when I do this, then no messages gets through.

The extractor seems to parse fine using the “Try” button;

 ![billede](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/924a76380bf391d36abaae7bd2fb95d47f37d333.png)

I have looked at some of the topics regarding this. There’s some key name char restrictions. Does not seems to be my problem, then theres some date parsing. I dont know if my time field is parsed as a date object, and theres a problem here?

Can anyone please help me how to troubleshot this? Or maybe even a solution.  
Thanks a lot.

And great product by the way:-)

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 7, 2018, 1:16pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/2 "2018-12-07T13:16:34Z")

</div>

Well, first, don’t select “Flatten” - that just tries to stuff it all into a single field with a weird format; so uncheck that. Then there’s the issue that it may not want to work after all due to the JSON object also containing a field named “message”, and I’m not sure how that plays along with Graylog JSON extractor (especially in copy mode).

An alternative option is to do this in a pipeline, a bit more work but if you create a rule as follows:

```
rule "parse the json log entries"
when
  true
then
  let json_tree = parse_json(to_string($message.message));
  let json_fields = select_jsonpath(json_tree, { time: "$.time", level: "$.level", message: "$.message", CorrelationID: "$.CorrelationID", SessionID: "$.SessionID", Component: "$.Component", ComponentVersion: "$.ComponentVersion", Action: " $.Action", Method: "$.Method", MemberID: "$.MemberID", TimeSpent: "$.TimeSpent" });

 set_field("level", to_string(json_fields.level));
 set_field("message", to_string(json_fields.message));
 set_field("CorrelationID", to_string(json_fields.CorrelationID));
 # etc. etc. etc.

 set_field(timestamp, flex_parse_date(json_fields.time));
 remove_field("time");
end

```

That will do the trick too - instead of setting each field individually you can also `set_fields(json_fields)` - but doing it individually means you can to the proper to\_string/to\_bool/to\_long/to\_double typecasting if it’s required. It will also grab the “time” field from your message, and will replace Graylog’s timestamp with the “proper” time of the event occuring since Graylog will not do that automatically for you.

(Also if you use a JSON extractor, you would still require a 2nd extractor to move the time field into the timestamp field)

---

<div class="post-metadata">

**Author:** ![mulgurul](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mulgurul](https://community.graylog.org/u/mulgurul)\
**Post date:** [December 10, 2018, 12:19pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/3 "2018-12-10T12:19:25Z")

</div>

Hi Ben  
First of all, a huge thanks for helping out on my issue. And even provide a complete example of a pipeline. Much appreciated…

I tried to submit the pipeline action that you have provided, but as a newbie stumbled upon some problems.

There was a few syntax errors. Two I could fix, but this one I’m not sure how to fix:

set\_field(timestamp, flex\_parse\_date(json\_fields.time));

I Tried to make timestamp to const string:

set\_field(“timestamp”, flex\_parse\_date(json\_fields.time));

But then got error: Expected type “string” for argument value but found object value in call to flex\_parse\_data etc… Can you hint me to a fix on this one?

Uncommenting that line and testing with raw string Json in message, it gives me an unchanged messages. I cant figure out what I’m doing wrong?

 ![billede](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/e/e0ef1ca1483ace031d11c948e754570177acb4a7.png)

Hope you can guide me here, thanks. //Peter

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 10, 2018, 10:46pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/4 "2018-12-10T22:46:51Z")

</div>

I think I made a typo somewhere, you’re right when you do a `set_field("timestamp", ...)` - not sure about the second error, I may have mistyped the name of the function (flex\_parse\_date), not sure if it’s actually called that…it’ll be in the documentation though. Other than that I’m not entirely sure - I’ve used this pattern in some of my own pipeline rules, but since I’m on vacation I can’t really check right now ☹

---

<div class="post-metadata">

**Author:** ![mulgurul](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mulgurul](https://community.graylog.org/u/mulgurul)\
**Post date:** [December 13, 2018, 8:55am UTC](https://community.graylog.org/t/failing-json-extractor/7944/5 "2018-12-13T08:55:55Z")

</div>

Hi again

I got it working, just needed some smaller adjustements,  
Allthough I’m wondering still why Json extractor couldn’t do the job.  
It shoudl be faster right, and somehow I like the idea of not having a dependency on a lot of hardcoded field names in a pipeline rule!

Best regards, Peter

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [December 13, 2018, 9:14am UTC](https://community.graylog.org/t/failing-json-extractor/7944/6 "2018-12-13T09:14:45Z")

</div>

> It shoudl be faster right, and somehow I like the idea of not having a dependency on a lot of hardcoded field names in a pipeline rule!

that is the reason that in Graylog 3.0 the function `set_fields` can be used to write the complete map into the key-value fields.

The nice about OSS software is that it can be extended by any user that has the need for something the initial developer did not thought about.

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 13, 2018, 9:36am UTC](https://community.graylog.org/t/failing-json-extractor/7944/7 "2018-12-13T09:36:45Z")

</div>

In the long run they’re both just as fast since it more or less does the same thing - and realistically speaking, a few microseconds more or less isn’t going to hurt 😉 Also, as @jan pointed out (and I keep pointing out his pointings out), in Graylog 3.0 (coming, as far as I recall, in February 2019) you can use set\_fields() with the direct output of parse\_json without the need for the intermediate select\_jsonpath.

Personally I prefer pipelines over extractors, mostly because you attach a pipeline to a stream which generally means you already “know” what you’ll be parsing. For example, our Graylog setup ingests about 2500msg/sec, and our heaviest pipeline only processes ~700 of those due to the stream routing being done before pipelines run, so we can tailor the pipelines very specifically to a subset of what we ingest.

To put that in perspective a little bit, everything comes in via Filebeat, about half of messages are generally formatted in syslog style lines, then we have about 33% with custom log formats that require additional parsing and generous use of lookup tables, the rest comes in from our microservice cluster, in various formats including a pure JSON format, where a pipeline “does the right thing” based on some fields being present (or absent) in a message.

We wouldn’t be able to run it as efficiently with only extractors. Personally speaking, I think pipelines are the more flexible feature, and while currently they do require some hardcoding of fields, you do have the benefit of knowing exactly what goes into your message once it’s parsed and processed.

---

<div class="post-metadata">

**Author:** ![mulgurul](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mulgurul](https://community.graylog.org/u/mulgurul)\
**Post date:** [December 14, 2018, 11:27am UTC](https://community.graylog.org/t/failing-json-extractor/7944/8 "2018-12-14T11:27:10Z")

</div>

Thanks for all your great and valuable feedback. I’m then continuing down the pipeline path, but have run into some other issues here. I’m having a little hard time to understand exactly what the pipeline does when the “Then” code has been executed. Does it just modify the original message so this modifications are visible when searching and showing the message in “search”?.

What happens is that my rule seems to be able to parse the Json in message field in the simulator, but when receiving messages from the real stream, it fails ALL messages. I took the message directly from a recent event, and tried it it the simulator. It was parsed fine.

Here’s an example of the data:

{ “time”: “2018-12-14 12:02:11.9210”, “level”: “INFO”, “message”: “DODP operation-entry: getBookmarks”, “CorrelationID”: “c472e04b-6ee2-47f1-9684-d354feb15d82”, “SessionID”: “UFbMbnX8i0a8BxqH6A\_s-g”, “Component”: “DodpMobile”, “ComponentVersion”: “1.0.1.0”, “Action”: “DODP-GETBOOKMARKS”, “MethodNameOrURL”: “/getBookmarks”, “MemberID”: “”, “BookNo”: null, “TimeSpentInMs”: “0”, “Hostname”: “BETADLWEB02”, “InitiatorIPAddress”: “192.168.0.7”, “UserAgent”: “DodpReader;1.0.40.25070”, “ResultCode”: null, “DataAsJson”: null }

Here’s the Rule:  
rule “parse the json log entries”  
when  
true  
then  
let json\_tree = parse\_json(to\_string(message.message)); let json\_fields = select\_jsonpath(json\_tree, { time: ".time", level: “.level", message: ".message”, CorrelationID: “.CorrelationID", SessionID: ".SessionID”, Component: “.Component", ComponentVersion: ".ComponentVersion”, Action: “.Action", ResultCode: ".ResultCode”, DataAsJson: “.DataAsJson" , InitiatorIPAddress: ".InitiatorIPAddress” , MethodNameOrURL: “.MethodNameOrURL", MemberID: ".MemberID”, UserAgent: “.UserAgent", TimeSpentInMs: ".TimeSpentInMs”, BookNo: “.BookNo", SourceLine: ".\_source\_line”, SourceMethod: “.\_source\_method", EventDateAtOrigin: ".EventDateAtOrigin” });

set\_field(“EventDateAtOrigin”, to\_date(json\_fields.EventDateAtOrigin));  
set\_field(“Level”, to\_string(json\_fields.Level));  
set\_field(“CorrelationID”, to\_string(json\_fields.CorrelationID));  
set\_field(“SessionID”, to\_string(json\_fields.SessionID));  
set\_field(“BookNo”, to\_string(json\_fields.BookNo));  
set\_field(“Hostname”, to\_string(json\_fields.Hostname));  
set\_field(“UserAgent”, to\_string(json\_fields.UserAgent));  
set\_field(“ResultCode”, to\_long(json\_fields.ResultCode));  
set\_field(“ComponentVersion”, to\_string(json\_fields.ComponentVersion));  
set\_field(“Component”, to\_string(json\_fields.Component));  
set\_field(“Action”, to\_string(json\_fields.Action));  
set\_field(“MemberID”, to\_string(json\_fields.MemberID));  
set\_field(“TimeSpentInMs”, to\_long(json\_fields.TimeSpentInMs));  
set\_field(“DataAsJson”, to\_string(json\_fields.DataAsJson));  
set\_field(“MethodNameOrURL”, to\_string(json\_fields.MethodNameOrURL));  
set\_field(“InitiatorIPAddress”, to\_string(json\_fields.InitiatorIPAddress));  
set\_field(“time”, to\_string(json\_fields.time));  
set\_field(“SourceLine”, to\_string(json\_fields.SourceLine));  
set\_field(“SourceMethod”, to\_string(json\_fields.SourceMethod));

set\_field(“timestamp”, parse\_date(substring(to\_string(json\_fields.time), 0, 23), “yyyy-MM-dd HH:mm:ss.SSS”));  
remove\_field(“time”);  
end

Simulator shows:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0ea977708ab8a1de8f7a2633f1e617b48364be0f.png)

Here’s where I see errors:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/a/a9fac84d85f840e3fe70212b2cbb011d4fe7819a.png)

Or at least think it means errors evem 0 error/s(1145) could be understood in many ways, anyway, theres no changes to the messages when I search. No new or modified fields. (Do I need to cycle index?)  
I dont know how to troubleshoot this. Do I need to look into Graylog server logs?

Sorry that I have to ask for more help…

//Peter

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 14, 2018, 1:53pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/9 "2018-12-14T13:53:58Z")

</div>

Hi Peter,

The rule looks good except for the select\_jsonpath, not sure if the forum software ate it, but you need to prefix the path with “$”.

And yes your view of the errors is correct, the throughput and errors are at 0/sec (probably because you aren’t putting anything through the stream), and over the lifetime of your pipeline it has picked up 1145 errors altogether.

Another reason it may error out is that the json parsing failed, I think that does show up in the main server log. You may need to take a look at that and throw some messages through and see what happens, and perhaps alter your rule after to have a more targeted condition - e.g. do a regex check for `"^\\{.*\}}$"` (keep in mind this is a little bit expensive to run) to ensure that the message field at least _looks_ like JSON before attempting to parse it.

Other than that I can’t really think of anything that would cause the errors unfortunately ☹

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [December 14, 2018, 2:29pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/10 "2018-12-14T14:29:31Z")

</div>

> The rule looks good except for the select\_jsonpath, not sure if the forum software ate it, but you need to prefix the path with “$”.

that is the reason you should use in this community to write code/rules

````auto
```
 your code / rule here
```
````

---

<div class="post-metadata">

**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\
**Post date:** [December 14, 2018, 3:54pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/11 "2018-12-14T15:54:32Z")

</div>

> [@benvanstaveren](#):
>
> Personally I prefer pipelines over extractors, mostly because you attach a pipeline to a stream which generally means you already “know” what you’ll be parsing.

We use different ports for different applications/functions, because it is easier to make stream about the input ID, you don’t need use multiple rules, or use regex about the hostnames. Also easier to debug the network traffic, and if you got a web server, you don’t need to administrate on graylog side, just set the 1234 port, and it will go the webserver stream.

But there is no good solution, it’s just my opinion.

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 14, 2018, 9:36pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/12 "2018-12-14T21:36:38Z")

</div>

To borrow a Perl-ism, TIMTOWTDI 🙂 We are in the “lucky” position that all our input is through Beats, and that we tag things with additional fields before they get to Graylog. Our stream rules are all exact-match on one field - except 3 streams that have 2 exact-match rules. So in general just as fast as checking the input ID 🙂

The reason we run it like that is that by way of the extra fields we know exactly which app is logging, and by extension the exact format it’s in, so we can tailor a pipeline exclusively to that (and extract additional data that we want).

I don’t really agree that there is “no” good solution, I’d like to rephrase that as “there are many good solutions, but which one is the best is subjective” 🙂

---

<div class="post-metadata">

**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\
**Post date:** [December 14, 2018, 9:45pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/13 "2018-12-14T21:45:14Z")

</div>

You are very lucky.  
We have more than 40 streams, 600+ sources, 50+ different source type  
Only the timestamp and the messages are the common fields in the logs. I’m lucky when the GL can phrase hostname.

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [December 14, 2018, 9:47pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/14 "2018-12-14T21:47:47Z")

</div>

Damn son… yeah, that gets complicated fast!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [December 28, 2018, 9:47pm UTC](https://community.graylog.org/t/failing-json-extractor/7944/15 "2018-12-28T21:47:58Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.