I can't receive logs in a graylog from java app running in wildfly server

Graylog Tech Challenges
1

Hi:
I have a java application running on a Wilfly 19 server.
The server is a Centos centos-release-8.2-2.2004.0.2.el8.x86_64.
I installed there a graylog 3.1 from docker-compose with Elastic 6.8.2 and mongo 3.

The graylog is running without any trouble. I configured the Wildfly standalone.xml file in order to write in the graylog, but I can’t receive any logs.

The docker-compose.yml file is like this:

version: '2'
services:
    # MongoDB: https://hub.docker.com/_/mongo/
    mongodb:
        image: mongo:3
        volumes:
            - mongo_data:/data/db
    # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
        volumes:
            - es_data:/usr/share/elasticsearch/data
        environment:
            - http.host=0.0.0.0
            - transport.host=localhost
            - network.host=0.0.0.0
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        mem_limit: 1g
    # Graylog: https://hub.docker.com/r/graylog/graylog/
    graylog:
        image: graylog/graylog:3.1
        volumes:
            - graylog_journal:/usr/share/graylog/data/journal
        environment:
            # CHANGE ME (must be at least 16 characters)!
            - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
            # Password: admin
            - GRAYLOG_ROOT_PASSWORD_SHA2=
            # - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
            - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.25.238:9000/      
            #- GRAYLOG_HTTP_BIND_ADDRESS=192.168.25.238:9000    
        links:
            - mongodb:mongo
            - elasticsearch
        depends_on:
            - mongodb
            - elasticsearch
        ports:
            # Graylog web interface and REST API
            - 9000:9000
            # Syslog TCP
            # - 1514:1514
            # Syslog UDP
            - 1514:1514/udp
            # GELF TCP
            # - 12201:12201
            # GELF UDP
            - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
    mongo_data:
        driver: local
    es_data:
        driver: local
    graylog_journal:
        driver: local

In the standalone.xml wildfly configuration file we have:

<root-logger>
                <level name="DEBUG"/>
                <handlers>
                    <handler name="CONSOLE"/>
                    <handler name="FILE"/>
                    <handler name="GelfLogger"/>
                </handlers>
            </root-logger>

<custom-handler name="GelfLogger" class="biz.paluch.logging.gelf.wildfly.WildFlyGelfLogHandler" module="biz.paluch.logging">
                <level name="DEBUG"/>
                <properties>
                    <property name="host" value="udp:localhost"/>
                   <!--  <property name="originHost" value="localhost"/>  -->
                    <property name="port" value="12201"/>
                    <property name="version" value="1.1"/>
                    <property name="extractStackTrace" value="true"/>
                    <property name="filterStackTrace" value="true"/>
                    <property name="includeFullMdc" value="true"/>
                    <property name="mdcProfiling" value="true"/>
                    <property name="maximumMessageSize" value="8192"/>
                </properties>
            </custom-handler>

I tried with almost same configuration in my local machine and it works properly. But in the server the value of externar uri is not localhost is the ip of the server. I am not sure if this is the cause of the problem. I can’t understand exactly what is happening.

Also I saw the logs of docker-compose in order to find some error there. I found this error message.

graylog_1        | 2021-06-30T01:54:59.185822126Z Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: **Unrecognized field "type" (class org.graylog.plugins.views.search.views.$AutoValue_ViewDTO$Builder), not marked as ignorable (11 known properties: "state", "search_id", "owner", "summary", "title", "_id", "description", "requires", "properties", "dashboard_state", "created_at"])**
graylog_1        | 2021-06-30T01:54:59.185829815Z  at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@3a64a122; pos: 37] (through reference chain: org.graylog.plugins.views.search.views.$AutoValue_ViewDTO$Builder["type"])

And also this other one:

graylog_1        | 2021-06-30T09:54:59.186420592Z 	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786) ~[graylog.jar:?]
graylog_1        | 2021-06-30T09:54:59.186423120Z 	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2181) ~[graylog.jar:?]
graylog_1        | 2021-06-30T09:54:59.186425677Z 	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
graylog_1        | 2021-06-30T09:54:59.186428252Z 	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
graylog_1        | 2021-06-30T09:54:59.186430904Z 	... 51 more
graylog_1        | 2021-06-30T17:54:59.186754917Z 2021-06-30 17:54:59,184 ERROR: org.graylog.plugins.views.search.db.SearchesCleanUpJob - Uncaught exception in periodical
graylog_1        | 2021-06-30T17:54:59.186783710Z java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
graylog_1        | 2021-06-30T17:54:59.186787929Z 	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
graylog_1        | 2021-06-30T17:54:59.186791221Z 	at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
graylog_1        | 2021-06-30T17:54:59.186794090Z 	at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
graylog_1        | 2021-06-30T17:54:59.186796921Z 	at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
graylog_1        | 2021-06-30T17:54:59.186799754Z 	at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar

I was reading about this error messages like here

and here:

And I tried to change to graylog v3.2
Now the error message of mongo did not appears. But I can’t see any log.

In the graylog admin console I active the input node: This is the info:

* bind_address:
0.0.0.0
* decompress_size_limit:
8388608
* number_worker_threads:
8
* override_source:
*<empty>*
* port:
12201
* recv_buffer_size:
262144

Any help or advice is very welcome!! Thanks in advance!

3

The problem was that I missed to add in the wildfly server the jar files for logstash-gelf
This files can be download from here:

https://logging.paluch.biz/download.html

Those dependencies must be located in:
wildfly-19.0.0.Final/modules/system/layers/base/

Then I gave permissions with chmod.
I restarted wildfly and logs messages started to appear.

One think to note is that problem appears in wildfly log and not in the graylog logs.

1 Like
4

Awesome news, Mariano! Thank you for posting your solution. Nicely done!

1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.