Input not Starting with java.lang.OutOfMemoryError: unable to create new native thread

etolampa · April 9, 2019, 8:58am

Hi,

I have a fresh installed graylog on debian but can’t get it working. It always gives me the “java.lang.OutOfMemoryError: unable to create new native thread” error on this route “/api/cluster/metrics/multiple”. I also can’t create any inputs.

I already checked my ulimit -u (http://docs.graylog.org/en/3.0/pages/faq.html#suddenly-parts-of-graylog-did-not-work-as-expected) it’s set to “1541132” that should be enough?

What else can I check? I also tried to set the JAVA Params from xms/xmx 1g to 2g bu got the same error.

Full Error:

Internal Server Error
Reason
java.lang.OutOfMemoryError: unable to create new native thread

Stacktrace
org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: unable to create new native thread
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.rethrow(GrizzlyHttpContainer.java:324)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.failure(GrizzlyHttpContainer.java:306)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:509)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$1(ProxiedResource.java:75)
at java.util.Optional.map(Optional.java:215)
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$2(ProxiedResource.java:75)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.graylog2.shared.rest.resources.ProxiedResource.getForAllNodes(ProxiedResource.java:74)
at org.graylog2.shared.rest.resources.ProxiedResource.getForAllNodes(ProxiedResource.java:69)
at org.graylog2.rest.resources.cluster.ClusterMetricsResource.multipleMetricsAllNodes(ClusterMetricsResource.java:74)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
… 14 more

macko003 · April 9, 2019, 1:18pm

First I suggest keep the settings at default until you know what you set.
Do you have free memory on your system? check free output.
Do you successfully started mongo and elasticsearch?
Could you write your server’s resources?
Does graylog works fine until you try to add an input?
What GL version do you try?

etolampa · April 9, 2019, 2:03pm

First I suggest keep the settings at default until you know what you set.
Yes. I just followed the guide here http://docs.graylog.org/en/3.0/pages/installation/os/debian.html and setup an reverse proxy via nginx to access the webinterface. http://docs.graylog.org/en/3.0/pages/configuration/web_interface.html#nginx

Do you have free memory on your system? check free output.

~# free -h
              total        used        free      shared  buff/cache   available
Mem:           6.0G        2.5G        2.9G        9.1M        578M          0B
Swap:          1.0G          0B        1.0G

Do you successfully started mongo and elasticsearch?

~# service mongodb status
● mongodb.service - An object/document-oriented database
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-09 08:05:38 UTC; 5h 49min ago
     Docs: man:mongod(1)
 Main PID: 222 (mongod)
    Tasks: 23 (limit: 127)
   CGroup: /system.slice/mongodb.service
           └─222 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

~# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-09 08:05:38 UTC; 5h 50min ago
     Docs: http://www.elastic.co
 Main PID: 266 (java)
    Tasks: 55 (limit: 127)
   CGroup: /system.slice/elasticsearch.service
           ├─266 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -
           └─672 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Could you write your server’s resources?
What do you exactly mean? It’s a vServer with 6 GB of RAM, 4 Cores + 200 GB disk space.

Does graylog works fine until you try to add an input?
I can use the Webinterface almost error free. If I open the network inspection tool in Chrome I see 500 Internal Server Error responses from “/api/cluster/metrics/multiple”.

What GL version do you try?
v3.0.1+de74b68

Thanks!

macko003 · April 9, 2019, 2:37pm

I suggest try it without nignx frontend.
The GL 3.0 has bugs with frontends, and we should be sure you don’t missconfigure anything in nginx. But I think it can’t cause OOM in graylog.

//Not related, but ES recommends disable swap for performance.

jan · April 9, 2019, 3:24pm

The GL 3.0 has bugs with frontends, and we should be sure you don’t missconfigure anything in nginx. But I think it can’t cause OOM in graylog.

Just to mention - 3.0.1 released which fixes the subfolder “issue” that was created with the new http_* settings.

@etolampa you have far more ressources as I use in my test environments - so that shouldn’t be a reason for the OOM. What java Version did you run with?

macko003 · April 10, 2019, 7:06am

OFF

Thanks, it’s new for me.

etolampa · April 10, 2019, 7:08am

@macko003 same error w/o nginx

@jan

~# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.212-b01, mixed mode)

system · April 24, 2019, 7:08am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
java.lang.OutOfMemoryError: unable to create new native thread Graylog Central (peer support)	5	2807	August 4, 2019
Inputs not Starting with java.lang.OutOfMemoryError: unable to create new native thread Graylog Central (peer support)	12	3256	October 11, 2017
Can't create new inputs Graylog Central (peer support)	7	1708	July 10, 2017
Graylog server extremely slow. cant get System/inputs Graylog Central (peer support)	19	6534	December 4, 2017
Graylogservice does not start because of java? Graylog Central (peer support)	12	2010	September 15, 2022

Input not Starting with java.lang.OutOfMemoryError: unable to create new native thread

Related topics