Graylog suddenly ceased to work

[tarkhil (Alex Povolotsky)]

Suddenly, after just adding memory (64 to 128 Gb) to server, Graylog ceased to work.

It’s a relatively fresh installation, no legacy, just elasticsearch upgraded 6.8.6 to 6.8.10 about three weeks ago without any trouble.

Now, graylog issues

2020-08-10 21:52:53,198 ERROR o.g.p.v.s.d.SearchesCleanUpJob [scheduled-0] Uncaught exception in periodical

java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
…
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field “type” (class org.graylog.plugins.views.search.views.$AutoValue_ViewDTO$Builder), not marked as ignorable (11 known properties: “state”, “search_id”, “owner”, “summary”, “title”, “_id”, “description”, “requires”, “properties”, “dashboard_state”, “created_at”])
at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@7f221170; pos: 37] (through reference chain: org.graylog.plugins.views.search.views.$AutoValue_ViewDTO$Builder[“type”])

After that, all API calls results in timeout. Elasticsearch cluster is green, plenty of memory, disks are not loaded, CPU is not loaded.

graylog-3.1.3

[tarkhil (Alex Povolotsky)]

I’d never could imagine the cause, or calculate it from log messages.

I’ve got my /etc/resolv.conf garbled, and resolving was not available. Fixing resolv.conf fixed a problem at once.