Report Generation Error

I’m trying to generate a report using the Enterprise 3.0 feature set. I get the following error from the UI (IPs removed):

{"type":"ApiError","message":"Timed out waiting for driver server to start.\nBuild info: version: 'unknown', revision: 'unknown', time: 'unknown'\nSystem info: host: '<HOST HERE REMOVED>', ip: '<IP HERE REMOVED>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-957.5.1.el7.x86_64', java.version: '1.8.0_191'\nDriver info: driver.version: unknown"}

I see the following error message in /var/log/graylog/server.log:


2019-02-15T11:50:12.632Z ERROR [OsProcess] org.apache.commons.exec.ExecuteException: Process exited with an error: 127 (Exit value: 127)

2019-02-15T11:50:12.745Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource

org.openqa.selenium.WebDriverException: Timed out waiting for driver server to start.

Build info: version: 'unknown', revision: 'unknown', time: 'unknown'

System info: host: '<HOST REMOVED>', ip: '<IP REMOVED>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-957.5.1.el7.x86_64', java.version: '1.8.0_191'

Driver info: driver.version: unknown

at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:202) ~[?:?]

at org.openqa.selenium.remote.service.DriverService.start(DriverService.java:188) ~[?:?]

at org.graylog.plugins.report.render.RemoteBrowserService.connectDriver(RemoteBrowserService.java:135) ~[?:?]

at org.graylog.plugins.report.render.RemoteBrowserService.connectDriver(RemoteBrowserService.java:111) ~[?:?]

at org.graylog.plugins.report.api.ReportService.fetchPdf(ReportService.java:259) ~[?:?]

at org.graylog.plugins.report.rest.ReportResource.generateReport(ReportResource.java:156) ~[?:?]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]

at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]

at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]

at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]

at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]

at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]

at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]

at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: org.openqa.selenium.net.UrlChecker$TimeoutException: Timed out waiting for [http://localhost:9515/status] to be available after 20000 ms

at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:100) ~[?:?]

at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:197) ~[?:?]

... 32 more

Caused by: java.util.concurrent.TimeoutException

at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_191]

at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:156) ~[graylog.jar:?]

at org.openqa.selenium.net.UrlChecker.waitUntilAvailable(UrlChecker.java:75) ~[?:?]

at org.openqa.selenium.remote.service.DriverService.waitUntilAvailable(DriverService.java:197) ~[?:?]

... 32 more

Note: If I monitor the system for port 9515 – I never see a listener brought up for the report generator.

Please use proper formatting - without that your posting is nearly impossible. So quick, edit your posting and surround the log message with ``` like described in the FAQ:

https://community.graylog.org/faq#format-markdown

That makes it more likely that someone can help you.

Formatting has been added per your request.

How did you install Graylog and the enterprise plugins?

I used the RPM distribution method of installation. I followed the CentOS installation instructions.

Hi Dustin,

Could you please verify that the folder indicated in the bin_dir option of your Graylog configuration file has the following binary files?

headless_shell
chromedriver
chromedriver_start.sh

Also make sure that those binaries have the right permissions (and owner) so that the user executing Graylog server can also execute them.

Hope that helps.

The local graylog server is running under the ‘graylog’ user. The permissions show read and execute for all users:

     4 -rwxr-xr-x. 1 root root       798 Feb 11 13:57 graylog-server
111944 -rwxr-xr-x. 1 root root 114627328 Feb 11 13:58 headless_shell
     4 -rwxr-xr-x. 1 root root       132 Feb 11 13:58 chromedriver_start.sh
  8368 -rwxr-xr-x. 1 root root   8565408 Feb 11 13:58 chromedriver

The directory is accessible for the graylog user account as well. Any other ideas?

Is that the bin_dir directory set in your Graylog server configuration file?

Also, could you try executing both the headless_shell and chromedriver binaries on a normal shell and share the output with us?

Sure thing – here you go:

./chromedriver: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory
[danders@ip-172-32-10-253 bin]$ ./headless_shell 
[0215/155618.653317:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale 
[0215/155618.656798:FATAL:zygote_host_impl_linux.cc(116)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux_suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
#0 0x000003efdfdf <unknown>
#1 0x000003e81a50 <unknown>
#2 0x00000515db40 <unknown>
#3 0x000003e34103 <unknown>
#4 0x0000051644b8 <unknown>
#5 0x000003e2d3a1 <unknown>
#6 0x000003f20ee8 <unknown>
#7 0x000003f20f75 <unknown>
#8 0x000003e6aa72 <unknown>
#9 0x7f0896c8b3d5 __libc_start_main
#10 0x00000259202a _start

Received signal 6
#0 0x000003efdfdf <unknown>
#1 0x000003efdb51 <unknown>
#2 0x7f089832f5d0 <unknown>
#3 0x7f0896c9f207 __GI_raise
#4 0x7f0896ca08f8 __GI_abort
#5 0x000003efc985 <unknown>
#6 0x000003e81ec2 <unknown>
#7 0x00000515db40 <unknown>
#8 0x000003e34103 <unknown>
#9 0x0000051644b8 <unknown>
#10 0x000003e2d3a1 <unknown>
#11 0x000003f20ee8 <unknown>
#12 0x000003f20f75 <unknown>
#13 0x000003e6aa72 <unknown>
#14 0x7f0896c8b3d5 __libc_start_main
#15 0x00000259202a _start
  r8: 00007f08989537c0  r9: 0000000000000002 r10: 0000000000000008 r11: 0000000000000206
 r12: 00007ffd7925d638 r13: 0000000000000161 r14: 00007ffd7925d640 r15: 00007ffd7925d648
  di: 0000000000001611  si: 0000000000001611  bp: 00007ffd7925cc90  bx: 00007ffd7925ccf0
  dx: 0000000000000006  ax: 0000000000000000  cx: ffffffffffffffff  sp: 00007ffd7925cb58
  ip: 00007f0896c9f207 efl: 0000000000000206 cgf: 0000000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.

I installed libX11 (yum install libX11) which resolved the chromedriver execution issue. However, here is the new error message I receive when trying to invoke a report from the UI:

{"type":"ApiError","message":"unknown error: Chrome failed to start: exited abnormally\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/share/graylog-server/bin/headless_shell is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=72.0.3583.1 (9ae827a6f6bbc9186d4e79681de885ce77d5bfe9),platform=Linux 3.10.0-957.5.1.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 67 milliseconds\nBuild info: version: 'unknown', revision: 'unknown', time: 'unknown'\nSystem info: host: '<HOST>', ip: '<IP>', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-957.5.1.el7.x86_64', java.version: '1.8.0_191'\nDriver info: driver.version: RemoteWebDriver"}

@unixfun looks like you are running a RedHat/CentOS system with a pretty old kernel. Either update to a 4.x kernel or disable reporting sandbox with: report_disable_sandbox = true
In the server configuration file.

Interesting note is that I used the default AWS 7.5 Centos Base image and it has a 3.x kernel. I upgraded the kernel to 4.2 and still received an error with headless_shell looking for libGLEXv2.so I installed chromium-libs-71.0.3578.98 after running yum whatprovides '*/libGLESv2.so*' and restarted graylog. With the addition of this package, I was able to successfully create a report.

I suggest additional Linux requirements be added to the documentation for future users. Thanks for the help!

Cool that it is working now for you! We will add more details to the documentation soon. The error regarding libGLES could be ignored, since in headless_mode no graphic acceleration is needed anyways.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.