Input not Starting with java.lang.OutOfMemoryError: unable to create new native thread

#1

Hi,

I have a fresh installed graylog on debian but can’t get it working. It always gives me the “java.lang.OutOfMemoryError: unable to create new native thread” error on this route “/api/cluster/metrics/multiple”. I also can’t create any inputs.

I already checked my ulimit -u (http://docs.graylog.org/en/3.0/pages/faq.html#suddenly-parts-of-graylog-did-not-work-as-expected) it’s set to “1541132” that should be enough?

What else can I check? I also tried to set the JAVA Params from xms/xmx 1g to 2g bu got the same error.

Full Error:

Internal Server Error
Reason
java.lang.OutOfMemoryError: unable to create new native thread

Stacktrace
org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: unable to create new native thread
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.rethrow(GrizzlyHttpContainer.java:324)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.failure(GrizzlyHttpContainer.java:306)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:509)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$1(ProxiedResource.java:75)
at java.util.Optional.map(Optional.java:215)
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$2(ProxiedResource.java:75)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.graylog2.shared.rest.resources.ProxiedResource.getForAllNodes(ProxiedResource.java:74)
at org.graylog2.shared.rest.resources.ProxiedResource.getForAllNodes(ProxiedResource.java:69)
at org.graylog2.rest.resources.cluster.ClusterMetricsResource.multipleMetricsAllNodes(ClusterMetricsResource.java:74)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
… 14 more

0 Likes

#2

First I suggest keep the settings at default until you know what you set.
Do you have free memory on your system? check free output.
Do you successfully started mongo and elasticsearch?
Could you write your server’s resources?
Does graylog works fine until you try to add an input?
What GL version do you try?

0 Likes

#3

First I suggest keep the settings at default until you know what you set.
Yes. I just followed the guide here http://docs.graylog.org/en/3.0/pages/installation/os/debian.html and setup an reverse proxy via nginx to access the webinterface. http://docs.graylog.org/en/3.0/pages/configuration/web_interface.html#nginx

Do you have free memory on your system? check free output.

~# free -h
              total        used        free      shared  buff/cache   available
Mem:           6.0G        2.5G        2.9G        9.1M        578M          0B
Swap:          1.0G          0B        1.0G

Do you successfully started mongo and elasticsearch?

~# service mongodb status
● mongodb.service - An object/document-oriented database
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-09 08:05:38 UTC; 5h 49min ago
     Docs: man:mongod(1)
 Main PID: 222 (mongod)
    Tasks: 23 (limit: 127)
   CGroup: /system.slice/mongodb.service
           └─222 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

~# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-09 08:05:38 UTC; 5h 50min ago
     Docs: http://www.elastic.co
 Main PID: 266 (java)
    Tasks: 55 (limit: 127)
   CGroup: /system.slice/elasticsearch.service
           ├─266 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -
           └─672 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Could you write your server’s resources?
What do you exactly mean? It’s a vServer with 6 GB of RAM, 4 Cores + 200 GB disk space.

Does graylog works fine until you try to add an input?
I can use the Webinterface almost error free. If I open the network inspection tool in Chrome I see 500 Internal Server Error responses from “/api/cluster/metrics/multiple”.

What GL version do you try?
v3.0.1+de74b68

Thanks! :slight_smile:

0 Likes

#4

I suggest try it without nignx frontend.
The GL 3.0 has bugs with frontends, and we should be sure you don’t missconfigure anything in nginx. But I think it can’t cause OOM in graylog.

//Not related, but ES recommends disable swap for performance.

0 Likes

(Jan Doberstein) #5

The GL 3.0 has bugs with frontends, and we should be sure you don’t missconfigure anything in nginx. But I think it can’t cause OOM in graylog.

Just to mention - 3.0.1 released which fixes the subfolder “issue” that was created with the new http_* settings.

@etolampa you have far more ressources as I use in my test environments - so that shouldn’t be a reason for the OOM. What java Version did you run with?

0 Likes

#6

OFF

Thanks, it’s new for me.

0 Likes

#7

@macko003 same error w/o nginx :frowning:

@jan

~# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b01-1~deb9u1-b01)
OpenJDK 64-Bit Server VM (build 25.212-b01, mixed mode)
0 Likes