I have encountered a new Lookup Table issue, and it’s not clear to me if the issue is a bug or a problem for me locally.
Problem: Some of my previously working lookup tables have ceased to function. A review of my server.log file complains that tables do not exist, but I can see them in the gui and in mongodb. I renamed a lookup table and updated the pipelines that consume it, but still get errors in the log.
Has anyone had this kind of issue with lookup tables? Any workarounds?
At this moment none of my tables appear to be working. I am going to completely delete and recreate a single table, cache and adapter to see if that helps.
This proved a little challenging - log files were cleared of useful information after 4-5 seconds of actual traffic.
While tailing the logs this time, I caught this message:
2018-07-10T15:20:52.761-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[LookupTableService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], OutputSetupService [TERMINATED], StreamCacheService [TERMINATED], PeriodicalsService [TERMINATED], ConfigurationEtagService [TERMINATED], KafkaJournal [TERMINATED], JerseyService [TERMINATED], BufferSynchronizerService [TERMINATED], JournalReader [TERMINATED]]}
2018-07-10T15:20:52.761-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[LookupTableService [STARTING]], STOPPING=[InputSetupService [TERMINATED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:740) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:312) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
I understand now that the LookupTableService is probably not stopping/starting properly.
Here are the logs from gl03 from when a restart is attempted.
Next steps:
Remove each of the non-default lookup tables one at a time, restarting all services between each. Maybe there is an issue with a particular table I can’t see.
Comb through Mongo and see if there any tiny differences.
We have a different problem that could be related. When we restart a Graylog node, we must go “Resume message processing” before the node will actually do any work. This is new to us - it didn’t work that way in the past.
