New-developed plugin causes graylog server to endlessly loop during boot

[mnl65]

A new function plugin developed by following the walk-through here causes graylog (2.3.0-alpha.3+c795033) to endlessly loop during the boot phase, as witnessed by the following log output:

2017-06-02T13:59:20.046+02:00 INFO  [CmdLineTool] Loaded plugin: Parse TAI64n timestamps 1.0.0 [com.anywi.plugins.parseTAI64n.ParseTAI64nPlugin]
2017-06-02T13:59:20.048+02:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.0-alpha.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-06-02T13:59:20.049+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.0-alpha.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-06-02T13:59:20.049+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.0-alpha.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-06-02T13:59:20.050+02:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.0-alpha.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-06-02T13:59:20.055+02:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.0-alpha.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-06-02T13:59:20.055+02:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.0-alpha.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-06-02T13:59:20.156+02:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-06-02T13:59:20.267+02:00 INFO  [Version] HV000001: Hibernate Validator null
2017-06-02T13:59:22.489+02:00 INFO  [CmdLineTool] Loaded plugin: Parse TAI64n timestamps 1.0.0 [com.anywi.plugins.parseTAI64n.ParseTAI64nPlugin]
2017-06-02T13:59:22.490+02:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.0-alpha.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-06-02T13:59:22.491+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.0-alpha.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-06-02T13:59:22.491+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.0-alpha.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-06-02T13:59:22.492+02:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.0-alpha.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-06-02T13:59:22.497+02:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.0-alpha.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-06-02T13:59:22.497+02:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.0-alpha.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-06-02T13:59:22.597+02:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-06-02T13:59:22.705+02:00 INFO  [Version] HV000001: Hibernate Validator null

This sequence is repeated every few seconds. BTW, before anyone points it out, the “Hibernate Validator null” line is also in the log files during a normal boot.

Graylog launches normally as soon as the offending plugin (Parse TAI64n timestamps 1.0.0) is removed.

When tested locally in the way described in the [blog post]((https://www.graylog.org/blog/71-writing-your-own-graylog-processing-pipeline-functions), everything works. The version used for local testing is 2.3.0-alpha.4-SNAPSHOT.

The log messages quoted above do not provide a lot of clues, so the question is: How can such a problem best be debugged - or even better, of course, any ideas about how to solve it?

[dennis]

That is very odd. Is the TAI64n plugin publicly available? I would like to check out the source and identify what could be causing this.

[mnl65]

Hi Dennis,

thanks for looking into this. The code is not available on any public site.

Further, the web interface (run with the command ./node/npm run start produces the following error message:

ERROR in /Users/xxxxxx/yyyyyy/graylog-project-repos/graylog-plugin-function-parseTAI64n/src/web/index.jsx
Module not found: Error: Can't resolve 'graylog-web-plugin/plugin' in '/Users/xxxxxx/yyyyyy/graylog-project-repos/graylog-plugin-function-parseTAI64n/src/web'
@ /Users/xxxxxx/yyyyyy/graylog-project-repos/graylog-plugin-function-parseTAI64n/src/web/index.jsx 7:14-50
@ multi (webpack)-dev-server/client?http://localhost:8080 webpack/hot/dev-server /Users/xxxxxx/yyyyyy/graylog-project-repos/graylog-plugin-function-parseTAI64n/src/web

The same error message is produced by the strlen plugin when following the instructions, and does not seem to affect the web interface, so probably not an issue. You never know, though.

[mnl65]

Progress, and potentially resolved:

Following the discussion here and here, the issue seems to be solved by setting

isolated=false

in the file graylog-plugin.properties.

The documentation states to set it to false only in case the plugin needs to work with other plugins. Since the function implemented did not depend on other function plugins, I left it as true like in the template.

Of course, knowing that the plugin processor is a plugin itself, it makes sense that it should be set to false.

It might be worth including more prominently in the documentation and maybe even set it to true in the function plugin example, since function plugins seem to be relevant only in conjunction with the pipeline processor - or is that too simplistic?