Generate Chart for non numeric fields generates errors


(Matt) #1

Hi,

For some of our dashboards I’ve created graphs that are just based on the count of the field over time. The graphs display ok but whilst looking into performance issues I identified many errors coming in from elasticsearch:

“Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [source], but got [text]”

This is the same for any count based graph against a non numeric field, e.g. keyword. I’ve rotated my write index which hasn’t helped (just incase I had a dodgy index but I didn’t expect it to be)

I’m running 2.4.0-beta.2+2364772 and have checked the release notes for 2.4.1 and 2.4.3 but haven’t seen this issue mentioned. I’ve been running the OVA as a PoC and haven’t upgraded yet as we plan to build a full production environment.

Any suggestions, expecially the blindingly obvious, are welcome!


(Jan Doberstein) #2

the statistics - or part of that can not be run on any kind of field - in this case the field is text and not numeric.

You could force special field types by creating a custom index mapping ( http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings )


(Matt) #3

Hey Jan, Thanks for the reply - I don’t need to change the index mapping, the source field is meant to be a text field. I’m just trying to use a Dashboard widget graph against the source value - In search, expand the source value and select ‘Generate Chart’ and then I see the following error in my elasticsearch logs:

2018-02-19_11:13:14.04842 [DEBUG][o.e.a.s.TransportSearchAction] [KB1YgBj] [graylog_13][3], node[KB1YgBj6Qe2FoadNzRLE_w], [P], s[STARTED], a[id=amd-F_2WS9OjF7zqDe3d4A]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[graylog_13, graylog_16, graylog_17, graylog_12, graylog_11], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[message], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=64, source={
2018-02-19_11:13:14.04892   "from" : 0,
2018-02-19_11:13:14.04950   "query" : {
2018-02-19_11:13:14.05022     "bool" : {
2018-02-19_11:13:14.05139       "must" : [
2018-02-19_11:13:14.05177         {
2018-02-19_11:13:14.05268           "query_string" : {
2018-02-19_11:13:14.05316             "query" : "source:<IP_Address>",
2018-02-19_11:13:14.05356             "fields" : [ ],
2018-02-19_11:13:14.05406             "use_dis_max" : true,
2018-02-19_11:13:14.05563             "tie_breaker" : 0.0,
2018-02-19_11:13:14.05600             "default_operator" : "or",
2018-02-19_11:13:14.05675             "auto_generate_phrase_queries" : false,
2018-02-19_11:13:14.05713             "max_determinized_states" : 10000,
2018-02-19_11:13:14.05752             "allow_leading_wildcard" : true,
2018-02-19_11:13:14.05997             "enable_position_increments" : true,
2018-02-19_11:13:14.06037             "fuzziness" : "AUTO",
2018-02-19_11:13:14.06414             "fuzzy_prefix_length" : 0,
2018-02-19_11:13:14.06416             "fuzzy_max_expansions" : 50,
2018-02-19_11:13:14.06635             "phrase_slop" : 0,
2018-02-19_11:13:14.06684             "escape" : false,
2018-02-19_11:13:14.06730             "split_on_whitespace" : true,
2018-02-19_11:13:14.06769             "boost" : 1.0
2018-02-19_11:13:14.06818           }
2018-02-19_11:13:14.06863         }
2018-02-19_11:13:14.06915       ],
2018-02-19_11:13:14.06964       "filter" : [
2018-02-19_11:13:14.07156         {
2018-02-19_11:13:14.07211           "bool" : {
2018-02-19_11:13:14.07282             "must" : [
2018-02-19_11:13:14.07319               {
2018-02-19_11:13:14.07406                 "range" : {
2018-02-19_11:13:14.07440                   "timestamp" : {
2018-02-19_11:13:14.08365                     "from" : "2018-02-12 11:13:08.132",
2018-02-19_11:13:14.08406                     "to" : "2018-02-19 11:13:08.132",
2018-02-19_11:13:14.08473                     "include_lower" : true,
2018-02-19_11:13:14.08510                     "include_upper" : true,
2018-02-19_11:13:14.08630                     "boost" : 1.0
2018-02-19_11:13:14.08653                   }
2018-02-19_11:13:14.09056                 }
2018-02-19_11:13:14.09094               }
2018-02-19_11:13:14.09165             ],
2018-02-19_11:13:14.09203             "disable_coord" : false,
2018-02-19_11:13:14.09266             "adjust_pure_negative" : true,
2018-02-19_11:13:14.09580             "boost" : 1.0
2018-02-19_11:13:14.09615           }
2018-02-19_11:13:14.09673         }
2018-02-19_11:13:14.09981       ],
2018-02-19_11:13:14.10017       "disable_coord" : false,
2018-02-19_11:13:14.10075       "adjust_pure_negative" : true,
2018-02-19_11:13:14.10112       "boost" : 1.0
2018-02-19_11:13:14.10144     }
2018-02-19_11:13:14.10388   },
2018-02-19_11:13:14.10476   "aggregations" : {
2018-02-19_11:13:14.10499     "gl2_histogram" : {
2018-02-19_11:13:14.10592       "date_histogram" : {
2018-02-19_11:13:14.10619         "field" : "timestamp",
2018-02-19_11:13:14.10656         "interval" : "1h",
2018-02-19_11:13:14.11169         "offset" : 0,
2018-02-19_11:13:14.11457         "order" : {
2018-02-19_11:13:14.11973           "_key" : "asc"
2018-02-19_11:13:14.12070         },
2018-02-19_11:13:14.12116         "keyed" : false,
2018-02-19_11:13:14.12159         "min_doc_count" : 0
2018-02-19_11:13:14.12376       },
2018-02-19_11:13:14.12424       "aggregations" : {
2018-02-19_11:13:14.12651         "gl2_stats" : {
2018-02-19_11:13:14.12727           "stats" : {
2018-02-19_11:13:14.12838             "field" : "source"
2018-02-19_11:13:14.12876           }
2018-02-19_11:13:14.13163         }
2018-02-19_11:13:14.13201       }
2018-02-19_11:13:14.13292     }
2018-02-19_11:13:14.13327   }
2018-02-19_11:13:14.13405 }}] lastShard [true]
2018-02-19_11:13:14.13610 org.elasticsearch.transport.RemoteTransportException: [KB1YgBj][127.0.0.1:9300][indices:data/read/search[phase/query]]
2018-02-19_11:13:14.13975 Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [source], but got [text]
2018-02-19_11:13:14.14016       at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.numericField(ValuesSourceConfig.java:306) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14083       at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.originalValuesSource(ValuesSourceConfig.java:289) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14123       at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:246) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14181       at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14218       at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14617       at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:210) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14825       at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:78) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14851       at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.14939       at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator.<init>(DateHistogramAggregator.java:71) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15036       at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorFactory.createAggregator(DateHistogramAggregatorFactory.java:80) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15065       at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorFactory.doCreateInternal(DateHistogramAggregatorFactory.java:74) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15098       at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorFactory.doCreateInternal(DateHistogramAggregatorFactory.java:37) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15200       at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:55) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15243       at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15519       at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15557       at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15657       at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15691       at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.15961       at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16000       at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16067       at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16104       at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16189       at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) [elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16231       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16346       at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.3.jar:5.6.3]
2018-02-19_11:13:14.16390       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
2018-02-19_11:13:14.16450       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
2018-02-19_11:13:14.16487       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Any ideas based on the above please?


(Matt) #4

Happy to supply more information if needed, to clarify:

Search for a message, select a unique non-numeric field such as the syslog ‘Source’ and choose ‘Generate Chart.’

Even generating this chart on my install creates the above error and slows performance down noticeably.

Any advice gratefully received.


(Jochen) #5

Please upgrade to the latest stable versions of Graylog (2.4.3 as of the time of writing) and rotate the active write index (System/Indices/Index Set/Maintenance). Then check if the problem still occurs.


(Matt) #6

Hey jochen, thanks for the response, I’ll give that a go and advise further.


(Matt) #7

Hey jochen, - I upgraded to 2.4.3 with no problems, however the issue remains. The same actions produce the error and the existing dashboards also still produce it.

Any advice welcomed.


(Jochen) #8

What’s the index mapping of your indices?
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-get-mapping.html

# curl 'http://localhost:9200/_mapping?pretty'

(Matt) #9

Hey jochen, I can’t send the whole lot due to the character limit, so this is the current index minus a bunch of the winlogbeat fields but the ‘source’ field is covered which is the primary culprit.

     graylog_18" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "store_generic" : {
              "match" : "*",
              "mapping" : {
                "index" : "not_analyzed"
              }
            }
          }
        ],
        "properties" : {
          "Action" : {
            "type" : "keyword"
          },
          "Alliance_Data_Bit9EndpointVisibility" : {
            "type" : "keyword"
          },
          "Alliance_Link_Bit9EndpointVisibility" : {
            "type" : "keyword"
          },
          "Alliance_Score_Bit9EndpointVisibility" : {
            "type" : "keyword"
          },
          "Alliance_Updated_Bit9EndpointVisibility" : {
            "type" : "date"
          },
          "BASE10NUM" : {
            "type" : "keyword"
          },
          "Client_IP" : {
            "type" : "keyword"
          },
          "Client_Username" : {
            "type" : "keyword"
          },
          "Comms_IP" : {
            "type" : "keyword"
          },
          "Company_Name" : {
            "type" : "keyword"
          },
          "DATE" : {
            "type" : "keyword"
          },
          "DATE_EU" : {
            "type" : "keyword"
          },
          "Description" : {
            "type" : "keyword"
          },
          "DestIP" : {
            "type" : "keyword"
          },
          "DestPort" : {
            "type" : "keyword"
          },
          "EvtLevel" : {
            "type" : "keyword"
          },
          "FPAction" : {
            "type" : "keyword"
          },
          "FPBytes_In" : {
            "type" : "keyword"
          },
          "FPBytes_In_Num" : {
            "type" : "long"
          },
          "FPBytes_Out" : {
            "type" : "keyword"
          },
          "FPBytes_Out_Num" : {
            "type" : "long"
          },
          "FPCategory" : {
            "type" : "keyword"
          },
          "FPDisposition" : {
            "type" : "keyword"
          },
          "FPDst_Host" : {
            "type" : "keyword"
          },
          "FPDst_IP" : {
            "type" : "keyword"
          },
          "FPDst_Port" : {
            "type" : "keyword"
          },
          "FPDuration" : {
            "type" : "keyword"
          },
          "FPDuration_Numb" : {
            "type" : "long"
          },
          "FPHTTP_Content_Type" : {
            "type" : "keyword"
          },
          "FPHTTP_Method" : {
            "type" : "keyword"
          },
          "FPHTTP_Proxy_Status_Code" : {
            "type" : "keyword"
          },
          "FPHTTP_Response" : {
            "type" : "keyword"
          },
          "FPHostIP" : {
            "type" : "keyword"
          },
          "FPLDAP_Connection" : {
            "type" : "keyword"
          },
          "FPPolicy" : {
            "type" : "keyword"
          },
          "FPReason" : {
            "type" : "keyword"
          },
          "FPRole" : {
            "type" : "keyword"
          },
          "FPSeverity" : {
            "type" : "keyword"
          },
          "FPSrc_Host" : {
            "type" : "keyword"
          },
          "FPSrc_Port" : {
            "type" : "keyword"
          },
          "FPURL" : {
            "type" : "keyword"
          },
          "FPUser" : {
            "type" : "keyword"
          },
          "FPUser_Agent" : {
            "type" : "keyword"
          },
          "FPUser_OU" : {
            "type" : "keyword"
          },
          "FPVersion" : {
            "type" : "keyword"
          },
          "FWDate" : {
            "type" : "date"
          },
          "FWTime" : {
            "type" : "keyword"
          },
          "File_Version" : {
            "type" : "keyword"
          },
          "First_Seen" : {
            "type" : "date"
          },
          "Group" : {
            "type" : "keyword"
          },
          "HOSTNAME" : {
            "type" : "keyword"
          },
          "HOUR" : {
            "type" : "keyword"
          },
          "HTTP_Status" : {
            "type" : "keyword"
          },
          "Host" : {
            "type" : "keyword"
          },
          "Hostname" : {
            "type" : "keyword"
          },
          "ID" : {
            "type" : "keyword"
          },
          "IPV4" : {
            "type" : "keyword"
          },
          "ISO8601_TIMEZONE" : {
            "type" : "keyword"
          },
          "Interface_IP" : {
            "type" : "keyword"
          },
          "Last_Update" : {
            "type" : "date"
          },
          "MINUTE" : {
            "type" : "keyword"
          },
          "MONTHDAY" : {
            "type" : "keyword"
          },
          "MONTHNUM" : {
            "type" : "keyword"
          },
          "Method" : {
            "type" : "keyword"
          },
          "PROTOCOL" : {
            "type" : "keyword"
          },
          "Port" : {
            "type" : "keyword"
          },
          "Process_Guid" : {
            "type" : "keyword"
          },
          "Process_MD5" : {
            "type" : "keyword"
          },
          "Process_Name" : {
            "type" : "keyword"
          },
          "Process_Path" : {
            "type" : "keyword"
          },
          "Product" : {
            "type" : "keyword"
          },
          "Product_Name" : {
            "type" : "keyword"
          },
          "Product_Version" : {
            "type" : "keyword"
          },
          "Protocol_Substatus" : {
            "type" : "keyword"
          },
          "Reason" : {
            "type" : "keyword"
          },
          "Referrer" : {
            "type" : "keyword"
          },
          "Result" : {
            "type" : "keyword"
          },
          "SECOND" : {
            "type" : "keyword"
          },
          "SEP_event_action" : {
            "type" : "keyword"
          },
          "SEP_event_app" : {
            "type" : "keyword"
          },
          "SEP_event_begin" : {
            "type" : "keyword"
          },
          "SEP_event_category" : {
            "type" : "keyword"
          },
          "SEP_event_category_code" : {
            "type" : "keyword"
          },
          "SEP_event_description" : {
            "type" : "keyword"
          },
          "SEP_event_direction" : {
            "type" : "keyword"
          },
          "SEP_event_domain" : {
            "type" : "keyword"
          },
          "SEP_event_end" : {
            "type" : "keyword"
          },
          "SEP_event_localhost_id" : {
            "type" : "keyword"
          },
          "SEP_event_localhost_ip" : {
            "type" : "keyword"
          },
          "SEP_event_localhost_port" : {
            "type" : "keyword"
          },
          "SEP_event_location" : {
            "type" : "keyword"
          },
          "SEP_event_message" : {
            "type" : "keyword"
          },
          "SEP_event_protocol" : {
            "type" : "keyword"
          },
          "SEP_event_quantity" : {
            "type" : "keyword"
          },
          "SEP_event_remote_hostname" : {
            "type" : "keyword"
          },
          "SEP_event_remote_id" : {
            "type" : "keyword"
          },
          "SEP_event_remote_ip" : {
            "type" : "keyword"
          },
          "SEP_event_remote_port" : {
            "type" : "keyword"
          },
          "SEP_event_remotefilepath" : {
            "type" : "keyword"
          },
          "SEP_event_rule" : {
            "type" : "keyword"
          },
          "SEP_event_server" : {
            "type" : "keyword"
          },
          "SEP_event_source" : {
            "type" : "keyword"
          },
          "SEP_event_user" : {
            "type" : "keyword"
          },
          "Search_Terms_v1" : {
            "type" : "keyword"
          },
          "Segment_ID" : {
            "type" : "keyword"
          },
          "Sensor_ID" : {
            "type" : "keyword"
          },
          "Server_IP" : {
            "type" : "keyword"
          },
          "Signed" : {
            "type" : "keyword"
          },
          "Size" : {
            "type" : "keyword"
          },
          "SrcIP" : {
            "type" : "keyword"
          },
          "SrcPort" : {
            "type" : "keyword"
          },
          "Start_Time" : {
            "type" : "date"
          },
          "SyslogPriority" : {
            "type" : "keyword"
          },
          "SyslogSeverity" : {
            "type" : "keyword"
          },
          "TIME" : {
            "type" : "keyword"
          },
          "TIMESTAMP_ISO8601" : {
            "type" : "date"
          },
          "Time_Taken" : {
            "type" : "keyword"
          },
          "Time_Taken_Numb" : {
            "type" : "long"
          },
          "Timestamp" : {
            "type" : "keyword"
          },
          "Type" : {
            "type" : "keyword"
          },
          "URI" : {
            "type" : "keyword"
          },
          "URI_Query" : {
            "type" : "keyword"
          },
          "User" : {
            "type" : "keyword"
          },
          "User_Agent" : {
            "type" : "keyword"
          },
          "Vendor" : {
            "type" : "keyword"
          },
          "WatchlistID" : {
            "type" : "keyword"
          },
          "Watchlist_ID" : {
            "type" : "keyword"
          },
          "Watchlist_Name" : {
            "type" : "keyword"
          },
          "Windows_Status" : {
            "type" : "keyword"
          },
          "YEAR" : {
            "type" : "keyword"
          },
          "collector_node_id" : {
            "type" : "keyword"
          },
          "connection_id" : {
            "type" : "long"
          },
          "connection_requests" : {
            "type" : "long"
          },
          "facility" : {
            "type" : "keyword"
          },
          "file" : {
            "type" : "keyword"
          },
          "from_gelf" : {
            "type" : "keyword"
          },
          "from_syslog" : {
            "type" : "keyword"
          },
          "full_message" : {
            "type" : "text",
            "analyzer" : "standard"
          },
          "gl2_remote_ip" : {
            "type" : "keyword"
          },
          "gl2_remote_port" : {
            "type" : "keyword"
          },
          "gl2_source_collector" : {
            "type" : "keyword"
          },
          "gl2_source_input" : {
            "type" : "keyword"
          },
          "gl2_source_node" : {
            "type" : "keyword"
          },
          "http_referer" : {
            "type" : "keyword"
          },
          "http_user_agent" : {
            "type" : "keyword"
          },
          "http_version" : {
            "type" : "keyword"
          },
          "icmpcode" : {
            "type" : "keyword"
          },
          "icmptype" : {
            "type" : "keyword"
          },
          "info" : {
            "type" : "keyword"
          },
          "input_type" : {
            "type" : "keyword"
          },
          "level" : {
            "type" : "long"
          },
          "message" : {
            "type" : "text",
            "analyzer" : "standard"
          },
          "millis" : {
            "type" : "float"
          },
          "name" : {
            "type" : "keyword"
          },
          "offset" : {
            "type" : "long"
          },
          "path" : {
            "type" : "keyword"
          },
          "remote_addr" : {
            "type" : "keyword"
          },
          "remote_user" : {
            "type" : "keyword"
          },
          "request_path" : {
            "type" : "keyword"
          },
          "request_verb" : {
            "type" : "keyword"
          },
          "response_bytes" : {
            "type" : "long"
          },
          "response_status" : {
            "type" : "long"
          },
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword",
            "fielddata" : true
          },
          "streams" : {
            "type" : "keyword"
          },
          "tags" : {
            "type" : "keyword"
          },
          "tcpack" : {
            "type" : "keyword"
          },
          "tcpflags" : {
            "type" : "keyword"
          },
          "tcpsyn" : {
            "type" : "keyword"
          },
          "tcpwin" : {
            "type" : "keyword"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "type" : {
            "type" : "keyword"
          },
          "winlogbeat_activity_id" : {
            "type" : "keyword"
          },
          "winlogbeat_computer_name" : {
            "type" : "keyword"
          },
          "winlogbeat_event_data_AccessList" : {
            "type" : "keyword"
          },
          "winlogbeat_event_data_AccessMask" : {
            "type" : "keyword"
          },
          "winlogbeat_event_data_ProcessID" : {
            "type" : "keyword"
          },

          },
          "winlogbeat_version" : {
            "type" : "long"
          }
        }
      }
    }
  },

Thanks in advance.


(Matt) #10

Hey - has anyone any idea on what may be causing this please? The performance impact is quite substantial and we may have to remove all the graphs that utilise the count for non-numeric fields which would be a massive shame as they are extremely useful.

Thanks in advance.


(Jan Doberstein) #11

Hej @mattmatics

just as an advice - for your GROK matches mark “named captures” only to not produce so many unwanted fields …

is the error only happening on the analyzed field source or others too?


(Matt) #12

Hey Jan, thank you for the advice, I have become aware of field bloat! I’ll apply that and see if helps.

We see it on the default ‘Source’ field or on any non numeric field - I have some converters running that cut into numeric fields and we don’t see the issue with them.


(Jochen) #13

@mattmatics I’ve tried to reproduce the issue you’ve described with a fresh installation of the Graylog OVA 2.4.3-1.

Unfortunately I was unable to reproduce it and everything works as expected. I’m able to create a chart, get quick values, and general field statistics for the “source” field:

For the record, the OVA is using Elasticsearch 5.6.3:

ubuntu@graylog:~$ curl 'http://192.168.1.145:9200/?pretty'
{
  "name" : "UX-eZ08",
  "cluster_name" : "graylog",
  "cluster_uuid" : "qg6OU8EiRUGSrve_-ZUTbA",
  "version" : {
    "number" : "5.6.3",
    "build_hash" : "1a2f265",
    "build_date" : "2017-10-06T20:33:39.012Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

The index templates also look normal:

ubuntu@graylog:~$ curl 'http://192.168.1.145:9200/_template?pretty'
{
  "graylog-internal" : {
    "order" : -1,
    "template" : "graylog_*",
    "settings" : {
      "index" : {
        "analysis" : {
          "analyzer" : {
            "analyzer_keyword" : {
              "filter" : "lowercase",
              "tokenizer" : "keyword"
            }
          }
        }
      }
    },
    "mappings" : {
      "message" : {
        "properties" : {
          "message" : {
            "type" : "text",
            "analyzer" : "standard",
            "fielddata" : false
          },
          "full_message" : {
            "type" : "text",
            "analyzer" : "standard",
            "fielddata" : false
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword",
            "fielddata" : true
          },
          "streams" : {
            "type" : "keyword"
          }
        },
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "store_generic" : {
              "match" : "*",
              "mapping" : {
                "index" : "not_analyzed"
              }
            }
          }
        ],
        "_source" : {
          "enabled" : true
        }
      }
    },
    "aliases" : { }
  }
}

And finally, the index mapping looks similar to the one you’ve posted (with regard to the “source” field):

ubuntu@graylog:~$ curl 'http://192.168.1.145:9200/_mapping?pretty'
{
  "graylog_0" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "store_generic" : {
              "match" : "*",
              "mapping" : {
                "index" : "not_analyzed"
              }
            }
          }
        ],
        "properties" : {
          "action" : {
            "type" : "keyword"
          },
          "controller" : {
            "type" : "keyword"
          },
          "facility" : {
            "type" : "keyword"
          },
          "from_gelf" : {
            "type" : "keyword"
          },
          "full_message" : {
            "type" : "text",
            "analyzer" : "standard"
          },
          "gl2_remote_ip" : {
            "type" : "keyword"
          },
          "gl2_remote_port" : {
            "type" : "keyword"
          },
          "gl2_source_input" : {
            "type" : "keyword"
          },
          "gl2_source_node" : {
            "type" : "keyword"
          },
          "http_method" : {
            "type" : "keyword"
          },
          "http_response_code" : {
            "type" : "long"
          },
          "ingest_time" : {
            "type" : "date"
          },
          "ingest_time_day" : {
            "type" : "long"
          },
          "ingest_time_epoch" : {
            "type" : "long"
          },
          "ingest_time_hour" : {
            "type" : "long"
          },
          "ingest_time_minute" : {
            "type" : "long"
          },
          "ingest_time_month" : {
            "type" : "long"
          },
          "ingest_time_second" : {
            "type" : "long"
          },
          "ingest_time_year" : {
            "type" : "long"
          },
          "level" : {
            "type" : "long"
          },
          "message" : {
            "type" : "text",
            "analyzer" : "standard"
          },
          "resource" : {
            "type" : "keyword"
          },
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword",
            "fielddata" : true
          },
          "streams" : {
            "type" : "keyword"
          },
          "ticks" : {
            "type" : "long"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "took_ms" : {
            "type" : "long"
          },
          "user_id" : {
            "type" : "long"
          }
        }
      }
    }
  }
}

The only hunch I still have is that using the pre-release versions of Graylog 2.4.0 and upgrading to a release version somehow corrupted the virtual machine (Elasticsearch?). In that case, setting Graylog up from scratch would “fix” the issue.

You could also try rotating the active write index and delete all old indices.


(Matt) #14

Hey jochen, Thank you very much for taking a look and trying to reproduce the issue, at lease I know that I should be ok if I re-install.

I think your analysis is correct on the issue probably being the 2.4.0 beta version in some fashion - the issue was present before any upgrade but the upgrade didn’t resolve. I needed the SSL fixes in the 2.4.0 version though but it has always been on the cards to build a full set up So I guess we can live with it.

I also compared all my mappings and templates to your and they appear to be identical as well, the only other thing I was looking at and wondering about was the aggregations - g12_histogram and g12_stats - could there be a weird definition (I don’t even know where or how they are defined tbh) that’s slipped in that could be forcing it to sum rather than total?

Again, thanks for all the assitance, it’s really appreciated.


(system) #15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.