I’ve been running Graylog successfully within Kubernetes for the past few months. Everything has been going really well and my small cluster is handling a decent volume of data. Thank you for building this product and releasing 3.1.1. Updating to version 3.1.1 has improved the UI performance and reduced load times on login. I’m just now getting around to addressing a couple configuration issues that I need guidance on.
I want to focus on the plugin setup. I understand there is a set of default plugins included with Graylog and these are located in the docker image under
/user/share/graylog/plugin directory. I also understand a user has the ability to define
GRAYLOG_PLUGIN_DIR to a specific directory outside of the default. The problem is when a user sets their own plugin directory the builtin plugins are no longer included. Which is expected since Graylog is no longer finding the include plugins within the user defined directory.
One option to solve this issue would be to copy over the default plugins into the users defined directory so Graylog could access those jars. I performed this operation for the included plugins but when I start Graylog the application fails to come online because the ThreatIntel plugin is unable to find a specific property. Below is the error logged on startup when the ThreatIntel plugin can be found in the user defined plugin directory.
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?] at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222] Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.graylog.plugins.threatintel.adapters.otx.OTXDataAdapter$Config$Builder, problem: Missing required properties: indicator apiUrl httpUserAgent at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@783dc0e7; pos: 413] (through reference chain: org.graylog2.lookup.dto.$AutoValue_DataAdapterDto$Builder["config"]) at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:277) ~[graylog.jar:?] at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1454) ~[graylog.jar:?] at com.fasterxml.jackson.databind.DeserializationContext.handleInstantiationProblem(DeserializationContext.java:1055) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapInstantiationProblem(BeanDeserializerBase.java:1667) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.finishBuild(BuilderBasedDeserializer.java:137) ~[graylog.jar:?]
In order for Graylog to start I have to remove this plugin from the directory. Once removed Graylog will come back online but now I have a number of warning related to Lookup tables and data adapters. Which I would assume is related to the fact I moved the plugin directory.
2019-09-17 17:11:34,479 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter whois of type whois, missing a factory. Is a required plugin missing? 2019-09-17 17:11:34,479 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter otx-api-domain of type otx-api, missing a factory. Is a required plugin missing? 2019-09-17 17:11:34,480 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter otx-api-ip of type otx-api, missing a factory. Is a required plugin missing? 2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table whois is referencing a missing data adapter 5cef4db5146bc60012abe85e, check if it started properly. 2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table otx-api-ip is referencing a missing data adapter 5cef4ded3f0a1c00129474e5, check if it started properly. 2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table otx-api-domain is referencing a missing data adapter 5cef4ded3f0a1c00129474e1, check if it started properly.
What would be the best way to get these plugins installed again and how could I correct the configurations within Graylog?
Thank you for your time and the Graylog product!