Missing data adapters and plugins

Graylog Central (peer support)
1

I’ve been running Graylog successfully within Kubernetes for the past few months. Everything has been going really well and my small cluster is handling a decent volume of data. Thank you for building this product and releasing 3.1.1. Updating to version 3.1.1 has improved the UI performance and reduced load times on login. I’m just now getting around to addressing a couple configuration issues that I need guidance on.

I want to focus on the plugin setup. I understand there is a set of default plugins included with Graylog and these are located in the docker image under /user/share/graylog/plugin directory. I also understand a user has the ability to define GRAYLOG_PLUGIN_DIR to a specific directory outside of the default. The problem is when a user sets their own plugin directory the builtin plugins are no longer included. Which is expected since Graylog is no longer finding the include plugins within the user defined directory.

One option to solve this issue would be to copy over the default plugins into the users defined directory so Graylog could access those jars. I performed this operation for the included plugins but when I start Graylog the application fails to come online because the ThreatIntel plugin is unable to find a specific property. Below is the error logged on startup when the ThreatIntel plugin can be found in the user defined plugin directory.

		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
	Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.graylog.plugins.threatintel.adapters.otx.OTXDataAdapter$Config$Builder, problem: Missing required properties: indicator apiUrl httpUserAgent
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@783dc0e7; pos: 413] (through reference chain: org.graylog2.lookup.dto.$AutoValue_DataAdapterDto$Builder["config"])
		at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:277) ~[graylog.jar:?]
		at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1454) ~[graylog.jar:?]
		at com.fasterxml.jackson.databind.DeserializationContext.handleInstantiationProblem(DeserializationContext.java:1055) ~[graylog.jar:?]
		at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapInstantiationProblem(BeanDeserializerBase.java:1667) ~[graylog.jar:?]
		at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.finishBuild(BuilderBasedDeserializer.java:137) ~[graylog.jar:?]

In order for Graylog to start I have to remove this plugin from the directory. Once removed Graylog will come back online but now I have a number of warning related to Lookup tables and data adapters. Which I would assume is related to the fact I moved the plugin directory.

2019-09-17 17:11:34,479 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter whois of type whois, missing a factory. Is a required plugin missing?
2019-09-17 17:11:34,479 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter otx-api-domain of type otx-api, missing a factory. Is a required plugin missing?
2019-09-17 17:11:34,480 WARN : org.graylog2.lookup.LookupTableService - Unable to load data adapter otx-api-ip of type otx-api, missing a factory. Is a required plugin missing?

2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table whois is referencing a missing data adapter 5cef4db5146bc60012abe85e, check if it started properly.
2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table otx-api-ip is referencing a missing data adapter 5cef4ded3f0a1c00129474e5, check if it started properly.
2019-09-17 17:11:34,708 WARN : org.graylog2.lookup.LookupTableService - Lookup table otx-api-domain is referencing a missing data adapter 5cef4ded3f0a1c00129474e1, check if it started properly.

What would be the best way to get these plugins installed again and how could I correct the configurations within Graylog?

Thank you for your time and the Graylog product!

2

to avoid issues in the future - if plugins are added to the default plugins or plugins moved into core - or plugins get a fix and you still link to the old version. My suggestion would be to use the volumen mount ability to link your custom plugins into the default path:

http://docs.graylog.org/en/3.1/pages/installation/docker.html#volume-mounted-plugin

3

Thank you for the reply. My understanding is changing the plugin directory is no longer advisable. The best solution would be to use the process you linked or download the plugins on each launch of the container to the default directory. Would this be correct?

Using kubernetes I would need to run a initContainer to populate that directory each time. Would it be possible to configure two different directories for Graylog to look into?

GRAYLOG_PLUGIN_DIR=/user/share/graylog/plugin:/var/lib/graylog/plugins

4

having multiple plugin path available in one or the other way is currently not supported. Please create a feature request for that over at Github:

1 Like
6

I opened an issue, thanks.

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.