Could not load quick values on field

(Drewmmiranda) #1

I suspect some bad data was input into my HTTPscstatus which normally ingests data via NXLog parsing IIS logs in standard format.

When I click “Quick values”, it returns error
Could not load quick values
Loading quick values failed with status: Error: cannot GET https://gl2web.ochsner.org/restapi/search/universal/absolute/terms?query=*&from=2017-02-15T05%3A40%3A24.000Z&to=2017-02-15T05%3A47%3A33.000Z&field=HTTPscstatus

If i go to the page directly in the web browser it returns
{"type":"ApiError","message":"Unable to fulfill search request"}

Any other field does return correctly. I checked the Graylog server logs (on all nodes, and web interface node) as well as the elasticsearch nodes, I don’t see any errors.

I imagine the field needs to be reindexed in elasticsearch but I’m not sure. Any ideas?

EDIT
The log on the web interface node DOES have an error, not very helpful though:

2017-02-16 16:40:09,835 WARN : org.graylog2.rest.resources.search.SearchResource - Unable to execute search: [reduce]

(Lennart Koopmann) #2

Hi Drew,

is the field of a numerical type? It will not work if it’s mapped as a string.

Make sure all the data you are sending into Graylog for this field is of a numerical type. You can have very strange effects if some messages have the field as a string and some have it in a numeric format. The first message sent into a new index (i.e. after an index is cycled and the deflector is switched) defines the type of the field in the index mapping. This is why you always want to have it in the same format (or define a custom mapping to enforce it)

Thanks,
Lennart

(Bill Murrin) #3

Another user and I had the same issue which we only observed after upgrading to 2.1.3.

Related group thread.
https://groups.google.com/forum/#!topic/graylog2/edYJ5oqSdYY

(Lennart Koopmann) #4

Can you share the full error you see in the Elasticsearch log file, together with the stacktrace? Usually, a reduce error will come with an explanation of what exactly went wrong. Thanks!

(Lennart Koopmann) #5

I might have just triggered the same bug. Reported here: https://github.com/Graylog2/graylog2-server/issues/3566

(Drewmmiranda) #6

Oh neat! I solved this on my end by creating a pipeline time to copy the
value to a new field.

I think what happened is that nxlog parsing a Windows IIS log, the
httpQuery field ended with a backslash for some reason causing it to run
together with the next field.

For example, the line below is OKAY
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem
cs-uri-query sc-status sc-bytes cs-bytes time-taken cs(User-Agent)
cs(Referrer) 2002-05-24 20:18:01 172.224.24.114 - 206.73.118.24 80 GET
/Default.htm - 200 7930 248 31
Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+2000+Server)
http://64.224.24.114/

BUT this line combines the query and status fields because the backslash
escapes the space character instead of the space being used as a delimiter
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem
cs-uri-query sc-status sc-bytes cs-bytes time-taken cs(User-Agent)
cs(Referrer) 2002-05-24 20:18:01 172.224.24.114 - 206.73.118.24 80 GET
/Default.htm query=data\ 200 7930 248 31
Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+2000+Server)
http://64.224.24.114/

All this this to say nxlog can do some wired things, elasticsearch can do
some weird things.

Glad you were able to reproduce why the particular bug may appear with
returning an error.