Lookup Table Issue - WARN <lookup table name> does not exist


(Dustin Tennill) #1

Hello,

I have encountered a new Lookup Table issue, and it’s not clear to me if the issue is a bug or a problem for me locally.

Problem: Some of my previously working lookup tables have ceased to function. A review of my server.log file complains that tables do not exist, but I can see them in the gui and in mongodb. I renamed a lookup table and updated the pipelines that consume it, but still get errors in the log.

I looked on github and found this post and responded as it looked very close to my issue.

Has anyone had this kind of issue with lookup tables? Any workarounds?

At this moment none of my tables appear to be working. I am going to completely delete and recreate a single table, cache and adapter to see if that helps.


(Jochen) #2

Please post the complete logs of your Graylog node(s) and the configuration of the lookup table and the lookup adapter and cache it’s based on:
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Dustin Tennill) #3

Here are config screenshots for one of the lookup tables with an issue.



Working on the logs now.


(Dustin Tennill) #4

All of the logs are full of: 2018-07-10T08:36:31.736-04:00 WARN [LookupTableService] Lookup table <banner_hosts_ip_lookuptable> does not exist

I will do a restart of everything and get clean logs before posting.


(Dustin Tennill) #5

This proved a little challenging - log files were cleared of useful information after 4-5 seconds of actual traffic.

While tailing the logs this time, I caught this message:

2018-07-10T15:20:52.761-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[LookupTableService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], OutputSetupService [TERMINATED], StreamCacheService [TERMINATED], PeriodicalsService [TERMINATED], ConfigurationEtagService [TERMINATED], KafkaJournal [TERMINATED], JerseyService [TERMINATED], BufferSynchronizerService [TERMINATED], JournalReader [TERMINATED]]}
2018-07-10T15:20:52.761-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[LookupTableService [STARTING]], STOPPING=[InputSetupService [TERMINATED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:740) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:312) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]

I understand now that the LookupTableService is probably not stopping/starting properly.

Here are the logs from gl03 from when a restart is attempted.
https://drive.google.com/file/d/1v72CTLzdTMvbK2d5HWJd9OA0ClKV9mL_/view?usp=sharing


(Jochen) #6

Please add all of this information to the GitHub issue you’ve linked to in your first post.


(Dustin Tennill) #7

That’s done. I am going to continue testing.

Next steps:
Remove each of the non-default lookup tables one at a time, restarting all services between each. Maybe there is an issue with a particular table I can’t see.
Comb through Mongo and see if there any tiny differences.

We have a different problem that could be related. When we restart a Graylog node, we must go “Resume message processing” before the node will actually do any work. This is new to us - it didn’t work that way in the past.


(Dustin Tennill) #8

LookupTables are up and running again for us.

In short: we had a failing non-used lookup table, and the problem didn’t hit us until we restarted services several days later.

Please review the github issue for all of the details.

We appreciate the assistance - this forum is excellent!!