JSON HTTP Stream is splittet into multiple messages

Graylog Central (peer support)
1

Hi,

1. Describe your incident:

i want to log a HTTP JSON Stream from my Bluecat DNS/DHCP Servers.

https://docs.bluecatnetworks.com/r/Address-Manager-Administration-Guide/DNS-query-event/9.3.0

I found online that i should use RAW/Plaintext TCP as input, which i select and get the messages in graylog.
But my problem is that one JSON Message from my Bluecat System was splittet into many single Messages in Graylog.

2. Describe your environment:
I use the actual Graylog 6.0.6 and new to graylog
.
3. What steps have you already taken to try and solve the problem?
I have no idea how can i map the messages together or tell graylog thats a JSON format and split the fields in one message.

I tried to use the “Null frame delimiter” in the input, but then i had many HTTP JSON Messages include http header in one graylog message.

4. How can the community help?
Question is how can i map the messages together or is there another input that i can usw. ore some other ideas to get a JSON HTTP Stream working as expected.

That some of the splittes of one Message

Messages
Messages2416×934 126 KB

Thats the JSON HTTP Message example
Example event message

{
    "dataType": "Message",
    "dataTypeId": 1,
    "key": "7bed8160-c8a1-4cb1-8a28-463909aa6844",
    "messageType": "AuthQuery",
    "messageTypeId": 1,
    "payloadType": "dnstap",
    "requestData": {
        "fullRcode": 0,
        "header": {
            "aa": false,
            "ad": true,
            "anCount": 0,
            "arCount": 1,
            "cd": false,
            "id": 3341,
            "nsCount": 0,
            "opcode": 0,
            "qdCount": 1,
            "qr": 0,
            "ra": false,
            "rcode": 0,
            "rd": true,
            "tc": false
        },
        "opt": {
            "do": false,
            "ednsVersion": 0,
            "extendedRcode": 0,
            "options": [
                {
                    "optCode": 10,
                    "optName": "Cookie",
                    "optValue": "hbbDFmHUM9w="
                }
            ],
            "udpPayloadSize": 4096
        },
        "question": [
            {
                "class": "IN",
                "domainName": "h1.example.com.",
                "questionType": "A",
                "questionTypeId": 1
            }
        ],
        "rcodeName": "NoError",
        "time": 1599780145572110343,
        "timePrecision": "ns"
    },
    "responseAddress": "127.0.0.1",
    "responsePort": 0,
    "serverId": "test-server",
    "serverVersion": "BIND 9.16.5",
    "socketFamily": "INET",
    "socketProtocol": "UDP",
    "sourceAddress": "127.0.0.1",
    "sourceId": "421bce7d-b4e6-b705-6057-7039628a9847",
    "sourcePort": 39207,
    "time": 1599780145572110343,
    "timePrecision": "ns"
}

Regardes

Andre

2

What you are seeing is the raw HTTP request data.

Unfortunately Graylog’s only current HTTP input is the GELF one. While it does accept json payloads, the GELF specifications mandates at a minimum a host field (or source instead) and a message field. If these fields are not present the GELF HTTP input will reject the message.

Are you able to ingest the logs another way? Possibly via syslog? Otherwise you’ll need something to accept your web request and format it with the proper GELF payload to send to a GELF http input. Any web server will do. The idea is to have the web server accept the JSON request and then either log the contents to text so it can be ingested with something like filebeat, or use a logging library to have the message sent directly to Graylog.

3

Hi,

thx for the response and the idea. I can´t change the format on the source device. I could try syslog and the whats happened, but for the load on the source http outputs are better. So i will check how can i use a webserver, input the stream and output it with filebeat.

Regardes

Andre

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.