Hi everyone,
I am currently configuring Graylog to collect execution status logs and event webhooks from an external application server. I have been attempting to stream HTTP JSON payloads directly into an HTTP input from this website, but the input intermittently drops incoming requests or throws unparsed payload errors in the server logs. I am trying to figure out whether this issue is caused by subtle header mismatches in the incoming web requests or if the Graylog HTTP input plugin requires specific content-type overrides to process the body correctly. On top of that, whenever a burst of requests hits the collector, the disk journal usage spikes rapidly and temporarily delays pipeline processing for our other active streams. Has anyone dealt with similar message drops or unparsed JSON buffers when receiving webhooks from third-party web platforms, and did tuning the input queue buffers or setting up custom extractors help stabilize the stream for you?
Hi there,
i had similar problems with receiving messages on http-input in graylog and did not get it to work.
but i did a workaround by starting a simple small python-script which is listening on a different port.
this script runs as systemd-service (simple-service) and only accepts messages, return 200 to sending server, adding correct source-ip to json, forwards body to graylog.
#!/usr/bin/python3
###################################################################################
# Small Python Log-Proxy for incomeing http -> add real-ip -> forward to graylog #
###################################################################################
##==##==##==##
import http.server
import json
import threading
import socket
import re
PYTHON_PORT = 9001
GRAYLOG_HOST = "<server-ip>"
GRAYLOG_PORT = <raw-or-json-input-port>
def clean_newlines(obj):
"""Rekursiv durch dict/list gehen und \r\n / \n in Strings entfernen bzw. ersetzen."""
if isinstance(obj, dict):
return {k: clean_newlines(v) for k, v in obj.items()}
elif isinstance(obj, list):
return [clean_newlines(v) for v in obj]
elif isinstance(obj, str):
return re.sub(r'\r?\n', ' ', obj)
else:
return obj
class Handler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers['Content-Length'])
body = self.rfile.read(length)
# get IP of Systems taht originally sends the message
client_ip = self.client_address[0]
## Get http_headers which can be added to json-message
event_type = str(self.headers['X-MY-HTTP-HEADER-I-NEED-TO-APPEND']) # in my case it was called x-event_type
## instantly return 200 OK for sending Server (http calls would run until timeout if not send 200)
self.send_response(200)
self.end_headers()
threading.Thread(target=self.forward, args=(body, client_ip, event_type)).start()
def forward(self, body, client_ip, event_type):
try:
data = json.loads(body)
data = clean_newlines(data)
data['source'] = client_ip
data['event_type'] = event_type
compact = json.dumps(data)
with socket.create_connection((GRAYLOG_HOST, GRAYLOG_PORT), timeout=5) as sock:
sock.sendall(compact.encode('utf-8') + b'\n')
print(f"Forward successful, source={client_ip}")
except Exception as e:
print(f"Forward failed: {e}")
http.server.HTTPServer(('0.0.0.0', PYTHON_PORT), Handler).serve_forever()
this solved my problem completely and i dont need to debug the http-input in graylog.
Hope this helps 
//Edit: i just created a github repo for this: GitHub - Coffee-IsLife/graylog_python_http-forwarder: Simple python-script which runs as systemd, accepts messages from third-party-http-calls (example gitlab-webhooks), return 200OK, adds correct original source-ip to json, adds headers i need to json, forwards http-body(json with added infos) to graylog-raw-input · GitHub
best regards,
Coffee_is_life