Graylog 3.3 failing to serve assets, Unexpected Token '<'

Hello,

I inherited a graylog instance that has been having problems for a couple months now. When I make a request to the server it often fails to find files on the server and I’m shown either a blank screen or certain components load. Usually continuing to refresh eventually fixes the problem, but there have been several times when it just wasn’t serving the files which created problems when trying to search for logs.

I suspect that it’s an issue with how the server has been configured, currently the setup is 3 graylog servers running in GCP behind a load balancer. The load balancer is set to INTERNAL_MANAGED, a policy of ROUND_ROBIN and the protocol is HTTP.

Versioning:

Graylog server version: Graylog 3.3.9+abab7dc
OS: CentOS Linux release 7.9.2009 (Core)
Mongo: db version v3.6.21
git version: 1cd2db51dce4b16f4bc97a75056269df0dc0bddb
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64

I parsed the server.log files on each server, there were no relevant logs to the issue. Looking through the server’s config file there doesn’t seem to be anything that’d be relevant, but if you have questions about specific fields I can post them.

After some further digging, it looks like what’s happening is when a call is made to some of the files behind the /assets/ URI, they sometimes return HTML instead of javascript. In the case of the login page, sometimes instead of /assets/builtins.7b2d519528c4e87cbc34.js returning the expected javascript file, it returns the same HTML that is served on /index.html. Since these HTML files start with <!DOCTYPE HTML> but they’re being treated like javascript, the console throws an Unexpected Token '<' error.

Real problem identified, opened new thread here: Graylog 3.3.9 serving index.html instead of requested file closing this thread.