Reporting error graylog v.5.0.2

1. Describe your incident and steps have you already taken:
Hello, since upgrade from graylog enterprise v4.3.9 to v5.0.2, generation reporting doesn’t work anymore.
I’ve set since v.3 the value report_disable_sandbox = true in server.conf.
The report scheduled generate segfault on lib libpthread-2.31.so.
I’ve read those topics but this doesn’t help me in my case :

2. Describe your environment:

  • OS Information: Debian 11.6

  • Package Version:

  • graylog-enterprise 5.0.2
  • elasticsearch 7.10.2
  • mongodb 6.0.3

Some interesting logs from graylog : (selenium/chromium related ?)

2023-01-11T15:23:19.580+01:00 ERROR [RemoteBrowserService] Failed to shutdown report render engine, process unavailable.
2023-01-11T15:23:19.580+01:00 ERROR [ReportRenderer] Unable to render report:
org.openqa.selenium.remote.UnreachableBrowserException: Error communicating with the remote browser. It may have died.
Build info: version: '4.5.0', revision: 'fe167b119a'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.0-19-amd64', java.version: '17.0.5'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [7a800630243fbd83434a5f2e690f9a05, getLog {type=browser}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 106.0.5249.119, chrome: {chromedriverVersion: 106.0.5249.119 (9f2101830b5..., userDataDir: /var/lib/graylog-server/rep...}, goog:chromeOptions: {debuggerAddress: localhost:34401}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: LINUX, proxy: Proxy(), se:cdp: ws://localhost:34401/devtoo..., se:cdpVersion: 106.0.5249.119, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 7a800630243fbd83434a5f2e690f9a05
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:571) ~[?:?]
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:602) ~[?:?]
    at org.openqa.selenium.remote.RemoteExecuteMethod.execute(RemoteExecuteMethod.java:40) ~[?:?]
    at org.openqa.selenium.remote.RemoteLogs.getRemoteEntries(RemoteLogs.java:81) ~[?:?]
    at org.openqa.selenium.remote.RemoteLogs.get(RemoteLogs.java:77) ~[?:?]
    at org.graylog.plugins.report.render.RemoteBrowserService.browserLogs(RemoteBrowserService.java:176) ~[?:?]
    at org.graylog.plugins.report.render.RemoteBrowserService.waitForRenderComplete(RemoteBrowserService.java:223) ~[?:?]
    at org.graylog.plugins.report.render.RemoteBrowserService.renderReport(RemoteBrowserService.java:130) ~[?:?]
    at org.graylog.plugins.report.render.ReportRenderer.renderReport(ReportRenderer.java:51) ~[?:?]
    at org.graylog.plugins.report.render.ReportRenderer.fetchPdf(ReportRenderer.java:66) ~[?:?]
    at org.graylog.plugins.report.scheduler.ReportRenderJob.lambda$doRun$0(ReportRenderJob.java:95) ~[?:?]
    at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [graylog.jar:?]
    at com.github.rholder.retry.Retryer.call(Retryer.java:160) [graylog.jar:?]
    at org.graylog.plugins.report.scheduler.ReportRenderJob.doRun(ReportRenderJob.java:105) [graylog-plugin-enterprise-5.0.2.jar:?]
    at org.graylog.plugins.report.scheduler.ReportRenderJob.run(ReportRenderJob.java:76) [graylog-plugin-enterprise-5.0.2.jar:?]
    at org.graylog.plugins.report.scheduler.ReportRenderSystemJob.execute(ReportRenderSystemJob.java:28) [graylog-plugin-enterprise-5.0.2.jar:?]
    at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.io.UncheckedIOException: org.asynchttpclient.exception.RemotelyClosedException: Remotely closed
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:73) ~[?:?]
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42) ~[?:?]
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56) ~[?:?]
    at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:49) ~[?:?]
    at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42) ~[?:?]
    at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56) ~[?:?]
    at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:99) ~[?:?]
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:181) ~[?:?]
    at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167) ~[?:?]
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142) ~[?:?]
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:547) ~[?:?]
    ... 23 more
Caused by: org.asynchttpclient.exception.RemotelyClosedException: Remotely closed
    at org.asynchttpclient.exception.RemotelyClosedException.INSTANCE(Unknown Source) ~[?:?]

If you use the enterprise version, it might be faster to open a ticket. At least I never used the reporting function, sorry.

Hello @John_Doe

I picked apart you logs.

2023-01-11T15:23:19.580+01:00 ERROR [RemoteBrowserService] Failed to shutdown report render engine, process unavailable.
2023-01-11T15:23:19.580+01:00 ERROR [ReportRenderer] Unable to render report:

Capabilities {acceptInsecureCerts: false,

RemotelyClosedException.INSTANCE(Unknown Source) 

I had that happen to myself a while back. It was this little guy here chromedriver_start.sh. I ensured it had the right permissions and it was present with these from that other post chromium, libx11-6, libx11-data, libx11-dev.

I did noticed this in the logs “Capabilities {acceptInsecureCerts: false”, by chnace do you have TCP/TLS enabled on this node?

Here are some of my personal notes when this happened. Not sure if it will help ya.
Using POSTFIX instead of sendmail. I did the following configuration( host IP address and interfaces)

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = localhost
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = ansible, ansible.domain.com, localhost.domain.com, localhost
relayhost =
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = 192.168.1.1
default_transport = smtp
relay_transport = smtp
inet_protocols = all
mynetworks = 192.168.1.100
root@ansible:/usr/local/bin#

Restart Postfix

Graylog enterprise version I had to install the following package

sudo apt-get install fontconfig fonts-dejavu

NOTE:

  • bin_dir = Directory with binaries needed for PDF generation
  • data_dir = Cache directory for PDF generation
  • report_disable_sandbox = Disables report generation sandbox

Hosts file configuration.

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.100 my.doamin.com

Unfortunately I was unable to send reports using the email provided. It did create the PDF file. I was able to send Notification alert to the provided email address configured. But could not send the report

Correct Graylog configuration file

# Email transport 
transport_email_enabled = true 
transport_email_hostname = 192.168.1.100 
transport_email_port = 25 
#transport_email_use_auth = true 
#transport_email_auth_username = you@example.com 
#transport_email_auth_password = secret 
transport_email_subject_prefix = [graylog] 
transport_email_from_email = anisble@doamin.com
transport_email_web_interface_url = https://192.168.1.100:9000

Hope that helps

1 Like

Hello,
I don’t have support feature because I use the entreprise free (under 2GB per day) @StefanAustin

Thanks for your reply @gsmith , it seems I’ve the required packets on my env :

# dpkg -l | grep "libx11\|chromium\|fonts-dejavu\|fontconfig"
ii  chromium                         108.0.5359.124-1~deb11u1       amd64        web browser
ii  chromium-common                  108.0.5359.124-1~deb11u1       amd64        web browser - common resources used by the chromium packages
ii  chromium-sandbox                 108.0.5359.124-1~deb11u1       amd64        web browser - setuid security sandbox for chromium
ii  fontconfig                       2.13.1-4.2                     amd64        generic font configuration library - support binaries
ii  fontconfig-config                2.13.1-4.2                     all          generic font configuration library - configuration
ii  fonts-dejavu                     2.37-2                         all          metapackage to pull in fonts-dejavu-core and fonts-dejavu-extra
ii  fonts-dejavu-core                2.37-2                         all          Vera font family derivate with additional characters
ii  fonts-dejavu-extra               2.37-2                         all          Vera font family derivate with additional characters (extra variants)
ii  libfontconfig1:amd64             2.13.1-4.2                     amd64        generic font configuration library - runtime
ii  libx11-6:amd64                   2:1.7.2-1                      amd64        X11 client-side library
ii  libx11-data                      2:1.7.2-1                      all          X11 client-side library
ii  libx11-dev:amd64                 2:1.7.2-1                      amd64        X11 client-side library (development headers)
ii  libx11-protocol-perl             0.56-7.1                       all          Perl module for the X Window System Protocol, version 11
ii  libx11-xcb1:amd64                2:1.7.2-1                      amd64        Xlib/XCB interface library

From mail configuration, all works like expected because I received the scheduled rapport in mailbox with this content :

Generating a report with the title “Rapport Daily” failed unexpectedly. Relevant error messages can be found in the report history and the server log.
Please contact your report author and/or the Graylog administrator for further assistance. Another attempt to generate this report will happen based on configured delivery frequencies.

When I try to exec chromderiver_start.sh with root :

# ./chromedriver_start.sh 
./chromedriver_start.sh: 11: : Permission denied

So I’ve set graylog user on bin dir but not better :

chown -R graylog. /usr/share/graylog-server/bin

For TLS part, I’ve http_enable_tls = false in server.conf.

Does anyone have the same behavior or a workaround ?

We are experiencing the same issue, and do not yet have a means to resolve it.

Hey ,

I found this.

Hello,

Thanks @gsmith for the link but for my case, this didn’t solved the problem.

However, I noticed an interesting behavior on the report generation.
When I generate a report with very little data, my report generates without any problem.
However, as soon as I try with more metrics, I get the following error after 30 seconds : {"type":"ApiError","message":"Unable to render report"}

It is like if the value report_generation_timeout_seconds was not taken into consideration or the timeout is caused by something else.

Hey

Oh so depend on the size of the report it will or will not get sent, is this correct?
Kind sound like a resource issue but im not 100% sure.

By chance did you double check if you have all your dependencies install? I think you showed some from the post above, just ensuring that was correct.
I also seen this in the doc’s, not sure if it will help.

report_generation_timeout_seconds
Default value: 180.

Time (in seconds) to wait for a report to load in the background.

To ensure all widgets in your report have time to fetch their data and load, Graylog waits up to the value set to this configuration option. When a report takes longer than configured to load, the report generation fails, and Graylog logs the error.

If reports in your Graylog setup do not generate, and the server displays a timeout error, you may need to increase this value.

Hi,

Yes, it’s correct.
From ressource side, server is provided with 6 core/12 GB ram but it’s challenged on CPU, I will add more core.

Dependencies seems to be OK for me

I tried to change this value but no success, always timeout around 30s

Hey

oh i see, I would look into why its not using the Default value: 180. This might be the issue.

Hey, so I found something here in the forum that i forgot to mention and have done before.
You should have a executable called headless_shell by executing that you may see you issue, just another trouble shooting idea.
Example

graylog@cbbe43b25f30:/usr/share/graylog-server/bin$ ./headless_shell
[0203/204157.475295:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale
[0203/204157.486690:FATAL:zygote_host_impl_linux.cc(116)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux_suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
#0 0x000003efdfdf <unknown>
#1 0x000003e81a50 <unknown>
#2 0x00000515db40 <unknown>
#3 0x000003e34103 <unknown>
#4 0x0000051644b8 <unknown>
#5 0x000003e2d3a1 <unknown>
#6 0x000003f20ee8 <unknown>
#7 0x000003f20f75 <unknown>
#8 0x000003e6aa72 <unknown>
#9 0x7fa18f0d7d90 <unknown>
#10 0x7fa18f0d7e40 __libc_start_main
#11 0x00000259202a _start

Received signal 6
#0 0x000003efdfdf <unknown>
#1 0x000003efdb51 <unknown>
#2 0x7fa18f0f0520 <unknown>
#3 0x7fa18f144a7c pthread_kill
#4 0x7fa18f0f0476 raise
#5 0x7fa18f0d67f3 abort
#6 0x000003efc985 <unknown>
#7 0x000003e81ec2 <unknown>
#8 0x00000515db40 <unknown>
#9 0x000003e34103 <unknown>
#10 0x0000051644b8 <unknown>
#11 0x000003e2d3a1 <unknown>
#12 0x000003f20ee8 <unknown>
#13 0x000003f20f75 <unknown>
#14 0x000003e6aa72 <unknown>
#15 0x7fa18f0d7d90 <unknown>
#16 0x7fa18f0d7e40 __libc_start_main
#17 0x00000259202a _start
  r8: 00007ffcd5262600  r9: 0000000000000002 r10: 0000000000000008 r11: 0000000000000246
 r12: 0000000000000006 r13: 0000000000000016 r14: 00007ffcd52630f0 r15: 00007ffcd52630f8
  di: 000000000000158d  si: 000000000000158d  bp: 000000000000158d  bx: 00007fa18f09f140
  dx: 0000000000000006  ax: 0000000000000000  cx: 00007fa18f144a7c  sp: 00007ffcd5262530
  ip: 00007fa18f144a7c efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]