No logs in docker elstaicsearch container

Hi all,
I have a problem where my elasticsearch docker crashes from time to time and i have to restart it all to get it going again.

When i log into the elasticsearch container and go to /var/log/elasticsearch/ i see no logs at all…

any help would be greatly appriciated

here is my docker-compose

Blockquote
version: ‘3’
services:
MongoDB: https://hub.docker.com/_/mongo/
mongo:
image: mongo:3
volumes:
- ./graylog/database/mongo_data:/data/db
networks:
- graylog
Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
networks:
- graylog
Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
user: $USER
image: graylog/graylog:3.1
volumes:
Mount local configuration directory into Docker container
- ./graylog/config:/usr/share/graylog/data/config
Mount local plugin file into Docker container
- ./graylog/plugin/:/usr/share/graylog/plugin/
Logs
- ./graylog/Logs/:/var/log/graylog-server/
environment:
CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://syslog.i4.dk:9000/api
- GRAYLOG_HTTP_EXTERNAL_URI=http://syslog.i4.dk:9000/
networks:
- graylog
dns:
- 8.8.8.8
- 8.8.4.4
depends_on:
- mongo
- elasticsearch
ports:
Graylog web interface and REST API
- 9000:9000
Syslog TCP
- 514:514
Syslog UDP
- 514:514/udp
GELF TCP
- 12201:12201
GELF UDP
- 12201:12201/udp
networks:
graylog:
driver: bridge
Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_journal:
driver: local
Blockquote

here is the output of my the docker container:

Blockquote
root@docker01:/docker# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db84d76b2660 graylog/graylog:3.1 “tini – /docker-ent…” 4 weeks ago Up 36 minutes (healthy) 0.0.0.0:514->514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:514->514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp docker_graylog_1
a9ba9a79f155 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2 “/usr/local/bin/dock…” 4 weeks ago Up 36 minutes 9200/tcp, 9300/tcp docker_elasticsearch_1
d5a2690b241a mongo:3 “docker-entrypoint.s…” 2 months ago Up 36 minutes 27017/tcp docker_mongo_1
Blockquote

Blockquote
[root@a9ba9a79f155 log]# ls
btmp grubby_prune_debug lastlog tallylog wtmp yum.log
Blockquote

as you can see im missing the elasticsearch folder?

can no one help? or are you not answering because im missing something simple or?

Hey @i4admin!

You mention the docker container is crashing but there are no elasticsearch logs. Are there any docker container logs indicating where the problem may lie?

1 Like

Why did i not think of that!!! now i feel stupid :slight_smile:
i didn’t note down the ID of the container last it ran so i have restarted the elsaticsearch continer and will wait for it to crash again and then see if i can find something in the docker logs.

but isn’t it wierd that there are elasticsearch logs inside the container?

I found this in the log when i started my elasticsearch container up again:

Blockquote
[2020-06-30T05:12:55,783][ERROR][o.e.ExceptionsHelper ] [ikLmw4q] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:280)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:270)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:176)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.base/java.lang.Thread.run(Thread.java:835)

im not sure if that could be the issue im facing?
I need to stress though that the server ran out of discspace so the error above could be related to that

@i4admin, I’m not able to help with that error message but perhaps you could find something relevant on the elasticsearch forums.

Here is a conversation about elasticsearch logging in docker: