Graylog Server fails to start after installation of plugin in plugin folder

I am new to this plugin world. I had to develop a custom plugin from the Twilio SMS plugin to suit my needs.

I built the plugin using “mvn package” and a jar is built.

I port the jar and install into the graylog server plugin folder.

I then run “sudo systemctl restart graylog-server.service” and service fails to start. It is only when I delete the plugin and I issue the restart it goes back to normal.

Is there a means to debug this? I see nothing in the logs as it fails to start up.

Please provide a copy of the log from boot to failure shutdown.

Hi @Ponet

Here are the logs: server was running and I then copied my plugin into plugin folder and I issued a restart call.

2020-03-26T16:50:24.407Z WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=SyslogUDPInput, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x567ab86e, L:/0:0:0:0:0:0:0:0%0:8514]) should be 262144 but is 425984.
2020-03-26T16:50:24.407Z WARN [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=SyslogUDPInput, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} (channel [id: 0x3af063b2, L:/0:0:0:0:0:0:0:0%0:8514]) should be 262144 but is 425984.
2020-03-26T16:50:24.408Z WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=NewBeatsForUSSDRICA, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x4c9c31f7, L:/0:0:0:0:0:0:0:0%0:5044]) should be 1048576 but is 425984.
2020-03-26T16:54:59.172Z WARN [LicenseChecker] License violation - Detected irregular traffic records
2020-03-26T16:55:57.658Z INFO [Server] SIGNAL received. Shutting down.
2020-03-26T16:55:57.666Z INFO [GracefulShutdown] Graceful shutdown initiated.
2020-03-26T16:55:57.666Z INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2020-03-26T16:56:01.669Z INFO [InputSetupService] Attempting to close input <org.graylog.plugins.beats.Beats2Input.5d13845da3ecc10c4329df38> [Beats].
2020-03-26T16:56:01.681Z INFO [InputSetupService] Input <org.graylog.plugins.beats.Beats2Input.5d13845da3ecc10c4329df38> closed. Took [12ms]
2020-03-26T16:56:01.682Z INFO [InputSetupService] Attempting to close input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5d13348da3ecc10a83436fa2> [Syslog UDP].
2020-03-26T16:56:01.684Z INFO [InputSetupService] Input <org.graylog2.inputs.syslog.udp.SyslogUDPInput.5d13348da3ecc10a83436fa2> closed. Took [2ms]
2020-03-26T16:56:01.694Z INFO [Buffers] Waiting until all buffers are empty.
2020-03-26T16:56:01.696Z INFO [Buffers] All buffers are empty. Continuing.
2020-03-26T16:56:01.697Z INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2020-03-26T16:56:01.704Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2020-03-26T16:56:01.704Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2020-03-26T16:56:01.704Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2020-03-26T16:56:01.704Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2020-03-26T16:56:01.705Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2020-03-26T16:56:01.705Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2020-03-26T16:56:01.705Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2020-03-26T16:56:01.705Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2020-03-26T16:56:01.706Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2020-03-26T16:56:01.704Z INFO [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks
2020-03-26T16:56:01.706Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2020-03-26T16:56:01.706Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2020-03-26T16:56:01.706Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2020-03-26T16:56:01.706Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2020-03-26T16:56:01.707Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2020-03-26T16:56:01.707Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2020-03-26T16:56:01.707Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2020-03-26T16:56:01.707Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2020-03-26T16:56:01.707Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.708Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2020-03-26T16:56:01.708Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.708Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2020-03-26T16:56:01.708Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.708Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2020-03-26T16:56:01.709Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2020-03-26T16:56:01.709Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2020-03-26T16:56:01.709Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.709Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2020-03-26T16:56:01.710Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2020-03-26T16:56:01.710Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2020-03-26T16:56:01.710Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2020-03-26T16:56:01.714Z INFO [GracefulShutdownService] Initiate shutdown for
2020-03-26T16:56:01.714Z INFO [GracefulShutdownService] Finished shutdown for , took 0 ms
2020-03-26T16:56:01.715Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2020-03-26T16:56:01.716Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2020-03-26T16:56:01.716Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseManagerPeriodical].
2020-03-26T16:56:01.716Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseManagerPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.718Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseReportPeriodical].
2020-03-26T16:56:01.718Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseReportPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.718Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.scheduler.ReportPeriodical].
2020-03-26T16:56:01.718Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.scheduler.ReportPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.719Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical].
2020-03-26T16:56:01.719Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] complete, took <0ms>.
2020-03-26T16:56:01.719Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.enterprise.search.db.SearchesCleanUpJob].
2020-03-26T16:56:01.719Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.enterprise.search.db.SearchesCleanUpJob] complete, took <0ms>.
2020-03-26T16:56:01.721Z INFO [GracefulShutdown] Goodbye.
2020-03-26T16:56:01.731Z INFO [JerseyService] Shutting down HTTP listener at <0.0.0.0:9000>
2020-03-26T16:56:01.738Z INFO [LogManager] Shutting down.
2020-03-26T16:56:01.739Z INFO [LookupDataAdapterRefreshService] Stopping 0 jobs
2020-03-26T16:56:01.775Z INFO [JournalReader] Stopping.
2020-03-26T16:56:01.793Z INFO [LogManager] Shutdown complete.
2020-03-26T16:56:01.804Z INFO [NetworkListener] Stopped listener bound to [0.0.0.0:9000]
2020-03-26T16:56:01.805Z INFO [ServiceManagerListener] Services are now stopped.

That log excerpt just shows the graylog server shutting down. That is not useful.

Please provide a copy of the log from startup to stopping due to failure.

Is there an option to set maximum debug level on Graylog server?

When I install my plugin and startup, the start up doesn’t continue. Hence I am unable to provide you with the logs you are looking for. Hence I would like to activate a max debug level.

You can change the logging levels of Graylog within the System > Logging menu within the Web UI.
Not sure if you can specify the logging level with the configuration file, check the docs for info on that.

I find it highly unlikely that nothing is written to the graylog-server log file when you start the process with the plugin enabled though.

Thanks. You have given me a few things to think about. I will check on them. I also find it strange that nothing is written to the graylog-server log file at start up.

You mention “with the plugin enabled though”. After generating the plugin with “mvn package” is there a further command I need to run in order to enable the plugin?

You mention “with the plugin enabled though”. After generating the plugin with “mvn package” is there a further command I need to run in order to enable the plugin?

No, you should build the jar file with mvn package and then copy it to your plugin folder and restart the service.

1 Like

Thanks all. I think I may have made some basic code errors. I am busy reading the plugin dev manual and checking.

I did a lot of renaming of my implementation so I think I need to adjust the relevant files in my code like in the properties file with plugin metadata.

I will post the findings on my developer mistakes.

1 Like

First mistake is that I didnt specify the plugin’s main method in the plugin file such that Graylog can determine how to startup my plugin.

I hope this is the only mistake I made.

Thank you all for your comments. Your comments guided me towards reading the manual and ultimately picking up my mistake. Yes it was the only mistake I made and everything works fine.

I will look to publish the code and share the plugin.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.