No logs in docker elstaicsearch container

i4admin · June 21, 2020, 1:23pm

Hi all,
I have a problem where my elasticsearch docker crashes from time to time and i have to restart it all to get it going again.

When i log into the elasticsearch container and go to /var/log/elasticsearch/ i see no logs at all…

any help would be greatly appriciated

here is my docker-compose

Blockquote
version: ‘3’
services:
MongoDB: Docker
mongo:
image: mongo:3
volumes:
- ./graylog/database/mongo_data:/data/db
networks:
- graylog
Elasticsearch: Install Elasticsearch with Docker | Elasticsearch Guide [6.7] | Elastic
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
networks:
- graylog
Graylog: Docker
graylog:
user: $USER
image: graylog/graylog:3.1
volumes:
Mount local configuration directory into Docker container
- ./graylog/config:/usr/share/graylog/data/config
Mount local plugin file into Docker container
- ./graylog/plugin/:/usr/share/graylog/plugin/
Logs
- ./graylog/Logs/:/var/log/graylog-server/
environment:
CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://syslog.i4.dk:9000/api
- GRAYLOG_HTTP_EXTERNAL_URI=http://syslog.i4.dk:9000/
networks:
- graylog
dns:
- 8.8.8.8
- 8.8.4.4
depends_on:
- mongo
- elasticsearch
ports:
Graylog web interface and REST API
- 9000:9000
Syslog TCP
- 514:514
Syslog UDP
- 514:514/udp
GELF TCP
- 12201:12201
GELF UDP
- 12201:12201/udp
networks:
graylog:
driver: bridge
Volumes for persisting data, see Volumes | Docker Docs
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_journal:
driver: local
Blockquote

here is the output of my the docker container:

Blockquote
root@docker01:/docker# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db84d76b2660 graylog/graylog:3.1 “tini – /docker-ent…” 4 weeks ago Up 36 minutes (healthy) 0.0.0.0:514->514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:514->514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp docker_graylog_1
a9ba9a79f155 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2 “/usr/local/bin/dock…” 4 weeks ago Up 36 minutes 9200/tcp, 9300/tcp docker_elasticsearch_1
d5a2690b241a mongo:3 “docker-entrypoint.s…” 2 months ago Up 36 minutes 27017/tcp docker_mongo_1
Blockquote

Blockquote
[root@a9ba9a79f155 log]# ls
btmp grubby_prune_debug lastlog tallylog wtmp yum.log
Blockquote

as you can see im missing the elasticsearch folder?

i4admin · June 30, 2020, 4:42am

can no one help? or are you not answering because im missing something simple or?

ttsandrew · June 30, 2020, 8:29pm

Hey @i4admin!

You mention the docker container is crashing but there are no elasticsearch logs. Are there any docker container logs indicating where the problem may lie?

i4admin · July 1, 2020, 4:46am

Why did i not think of that!!! now i feel stupid
i didn’t note down the ID of the container last it ran so i have restarted the elsaticsearch continer and will wait for it to crash again and then see if i can find something in the docker logs.

but isn’t it wierd that there are elasticsearch logs inside the container?

i4admin · July 1, 2020, 5:00am

I found this in the log when i started my elasticsearch container up again:

Blockquote
[2020-06-30T05:12:55,783][ERROR][o.e.ExceptionsHelper ] [ikLmw4q] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:280)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:270)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.base/java.lang.Thread.run(Thread.java:835)

im not sure if that could be the issue im facing?
I need to stress though that the server ran out of discspace so the error above could be related to that

ttsandrew · July 1, 2020, 11:58am

@i4admin, I’m not able to help with that error message but perhaps you could find something relevant on the elasticsearch forums.

Here is a conversation about elasticsearch logging in docker:

system · July 15, 2020, 11:58am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can't get docker graylog up and running Graylog Central (peer support)	3	6528	January 2, 2019
Docker: Can't open up streams and elasticsearch crashes Graylog Central (peer support)	12	2515	March 28, 2019
Having a look a graylog but it wont work Graylog Central (peer support)	2	1392	October 17, 2017
Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200) Graylog Central (peer support)	5	2270	April 25, 2020
Can't search anymore in graylog error cannot GET http: Graylog Central (peer support)	14	4144	February 13, 2019

No logs in docker elstaicsearch container

Related Topics