Capture Outbound Websocket connection JSON formatted data inside Graylog for processing, then upload via host text logs via filebeat

I have an application that creates a outbound Websocket connection to send JSON formatted data.

I’d like to capture that information inside of Graylog for processing, to relieve some client scripts I have that are processing and then uploading via text logs on the host (via filebeat).

Is there an input that can be used to have applications connect (via websocket) and post JSON formatted messages to be captured and logged?
I tried via the RAW/Plaintext, and I just get the HTTP request/response.

Thanks!

2021-05-24 17:20:12.072 +00:00	172.30.0.254
Connection: Upgrade
2021-05-24 17:20:12.072 +00:00	172.30.0.254
Sec-WebSocket-Key: fOpmqo2phXTtd6RsErtfdw==
2021-05-24 17:20:12.072 +00:00	172.30.0.254
Upgrade: websocket
2021-05-24 17:20:12.072 +00:00	172.30.0.254
GET / HTTP/1.1
2021-05-24 17:20:12.072 +00:00	172.30.0.254
User-Agent: WebSocket++/0.8.1

Hey, Brian,

Just a note. I noticed that your post hasn’t received a response. It’s an important question that deserves the community’s attention. So, let’s see if we can get more eyes on this. Have your looked at this previous post? It may be similar to your question (let me know):

https://community.graylog.org/t/logging-any-inbound-http-json-or-specific-format-required/8301/8

Also, here are a couple of other links that may help:

The more information you can provide the community, the more likely a response. Let me know if the additional help provide more context.

1 Like

I’m currently trying to re-write the websocket code to use http instead of https, because I think I’m getting only the scrambled https data, and of course, Graylog is just writing that as a log message, as described as “PlainText/Raw” should be.

Right now, the software I want to log messags from (trunk-recorder) initiates a outbound websocket connection attempt to a websocket based “statusServer”.
The same open source software author has a socket server, that accepts that connection and outputs the data to a webpage.
Based on some input from comments over there Send all calls [Encrypted/Not covered] to status server for Stats · Issue #462 · robotastic/trunk-recorder · GitHub, I adapted what I think would be a replacement, but its still based in HTTPS. Update index.js by kcwebby · Pull Request #1 · kcwebby/trunk-recorder-status-server · GitHub

I am a “script kid” and thus good with light programming, but heavy in sql and unix shell administrative scripts. So this coding requirement is all pretty well above my head. Especially Java.
I’m open to anything really to get this work, just thought the adaptation would be easy. As you read the comments, the other guy posted code that goes to Splunk. Of course, Splunk has a easy way to import the data even on HTTPS. (sorry)

I’d really appreciate any help or feedback. I’ll keep tinkering on my end and update this topic if I figure something out. But if someone else has something that might work, that could be adapted that would be great. I’ve pretty well outlined what I’m trying to log.

Once its in Graylog, I’m very comfortable with the extractors and pipelines needed to do what is needed to get the data useful.

Thanks again, in advance!

Hey @kcwebby,

Can you give us some more detail about your configuration? Re-writing the websocket code seems like a lot of effort for something we might be able to accomplish another way, which on its own wouldn’t bother me as much, but you’re also reducing channel security to accomplish the task which is very unfortunate.

Where is the JSON payload originating? Is it possible that Graylog could retrieve it via the JSON API endpoint listener?

What kind of Plaintext/RAW input are you using? If the problem is with decrypting the payload in Graylog then maybe it just needs some additional configuration. Have you tried setting up a TCP Plaintext/RAW input with support for client certificates?

3 Likes

Awesome participation, @ttsandrew ! You’re contributions are getting noticed in the community! Keep up the great work!

Thanks ttsandrew:

What I’m trying to do is capture the json status/log entries for statistics calcs into Graylog.
The software, Trunk-Recorder has a “statusServer” setting where it will establish a outbound websocket connection. Graylog is unable to accept this connection, as far as I can tell.

The rewrite of that websocket code is mostly a translator between the trunk-recorder software and Graylog. Heres a link to the documentation trunk-recorder/STATUS-JSON.md at master · robotastic/trunk-recorder · GitHub

I was able to get the code modified and use a Plaintext/RAW input, but its still less than ideal. Here’s a sample of what I logged when I connected the corrected http code from my post to the plaintext/raw.

IDEALLY: it would be great if you could have a websocket server as an input.

2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
POST / HTTP/1.1
2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
{"event":{"data":{"source":"1800455","position":"22.559999999999999","time":"1622133080","signal_system":"p25","emergency":"false","tag":""},"type":"unit"}}
2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
POST / HTTP/1.1
2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
Transfer-Encoding: chunked
2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
Authorization: Splunk <SPLUNKTOKEN>
2021-05-27 16:31:28.132 +00:00	108.xx.xx.215
9c