Failed to parse field "level" with string "notice"

Graylog Central (peer support)
1

Hi all, I just had a problem with parsing a firewall’s log. Please note that this configuration worked OK for over a week but today I noticed no logs were collected/parsed, with the following debug log:

[2019-02-04T15:48:02,491][DEBUG][o.e.a.b.TransportShardBulkAction] [06hPakH] [graylog_9][0] failed to execute bulk item (index) index {[graylog_deflector][message][df4b25c2-288b-11e9-9f41-005056893f91], source[{"date":"2019-02-04","level":"notice","gl2_remote_ip":"x.x.x.x","gl2_remote_port":18425,"streams":["000000000000000000000001"],"source":"x.x.x.x","message":"<189>date=2019-02-04 time=15:48:01 bla bla bla ... level=notice ... bla bla bla bla"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [level] of type [long]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:301) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:606) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:404) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96) ~[elasticsearch-6.5.4.jar:6.5.4]
        ...
        Caused by: java.lang.IllegalArgumentException: For input string: "notice"
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.toLong(AbstractXContentParser.java:199) ~[elasticsearch-x-content-6.5.4.jar:6.5.4]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:220) ~[elasticsearch-x-content-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:714) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:685) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:1050) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:295) ~[elasticsearch-6.5.4.jar:6.5.4]

As you can see, the “level” field would not be correctly parsed (elasticsearch was expecting type “long”, but it really is a string). The problem seems to be that the message field itself contained a “level” field, which I was extracting from a Raw UDP input with the following extractor:

{
  "extractors": [
    {
      "title": "Firewall log parser Base",
      "extractor_type": "grok",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "date=%{NOTSPACE:date} time=%{NOTSPACE:time} devname=%{NOTSPACE:devname}.* level=%{NOTSPACE:level}.*"
      },
    }
  ],
  "version": "2.5.1"
}

From the log above it seems that the Raw UDP already provide a “level” field, and my extractor was overloading it. Is this interpretation correct?

I solved the problem editing my extractor to remove the “level” field extraction, but I have some questions:

  • why the problem manifested only after 2 index rotation?
  • why the “Raw UDP” input itself uses the “level” field?
  • where I can find a list of field to avoid in my extractors?
  • how can I see all hidden fields?
  • am I missing something?

Thanks.

2

What is the problem? The ES can’t handle the “notice” as “long”. It’s normal.
Change “notice” to number, or change ES mapping, or use another fields instead of “level”.

3

Sure, “notice” is not of type long. What I did not understand is why, on index rotation, ES began treating “level” as of type long rather than string. Moreover, I can see an hidden “level” field in the UI (shown when selecting “all fields” in the search box).

So, does “level” really is an hidden, reserved field (similar to gl2_remote_ip and the likes)? Or something dirtied “level” at log rotation time?

Thanks.

4

level is not hidden. GL shows only the fields of the current search result.
I don’t know what do you change on your system what can cause this. Maybe an elasticsearch mapping can cause it. you can check it. Run this for thew old and the new index.

curl -H ‘Accept: application/json’ -XGET http://IP:9200/INDEX_NAME/_mapping?pretty

5

after the index rotation - if no template force a specific type of a field - the first message that is ingested into a field let Elasticsearch guess the field type.

So in your case @shodanshok the first message after rotation was a long and not a string. That means you have multiple sources using the same field name but with different types of data. You have now multiple options to solve.

  • create a custom elasticsearch mapping that force a specific type (string could hold everything but no math operations are possible)
  • create a processing pipeline that checks the content and place strings and numbers into different fields.
  • create a processing pipeline that translate the strings to numbers or visa-vie
6

You are right: ES mapping really defines “level” as “long”, and “level” is not hidden in any way.

I think I tracked what happened: the firewall log extractor define “level” as string/data, but another extractor (the syslog udp one) define “level” as number/long. I suppose that, on index rotation, a first syslog udp message, with numeric “level”, settled this field to be of type “long”. Subsequent messages from firewall had problem with their string representation of the “level” field.

The inverse was permitted: if “level” is of type string/data, then a long can be coverted in textual representation.

Does that make sense?
Thanks.

7

Hi Jan, so what you wrote confirmed my findings.
Well, it seems I need to more carefully check field type.
Thank you all.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.