Logging any inbound http json or specific format required?

Graylog Central (peer support)
1

Hi Community-
I was hoping it would be simple to set up an arbitrary input (I chose GELF HTTP) to be able to act at the very least as a heartbeat for a service calling in with webhooks (Mailgun API). Mailgun webhooks don’t allow port, so I tossed up an nginx proxy_pass, and that works as far as data flow goes. A demo/example inbound webhook message is:

event-data: {
  "tags": [
    "my_tag_1",
    "my_tag_2"
  ],
  "timestamp": 1521472262.908181,
  "storage": {
    "url": "https://se.api.mailgun.net/v3/domains/ihtspirit.com/messages/message_key",
    "key": "message_key"
  },
  "envelope": {
    "sending-ip": "209.61.154.250",
    "sender": "bob@ihtspirit.com",
    "transport": "smtp",
    "targets": "alice@example.com"
  },
  "recipient-domain": "example.com",
  "id": "CPgfbmQMTCKtHW6uIWtuVe",
  "campaigns": [

  ],
  "user-variables": {
    "my_var_1": "Mailgun Variable #1",
    "my-var-2": "awesome"
  },
  "flags": {
    "is-routed": false,
    "is-authenticated": true,
    "is-system-test": false,
    "is-test-mode": false
  },
  "log-level": "info",
  "message": {
    "headers": {
      "to": "Alice ",
      "message-id": "20130503182626.18666.16540@ihtspirit.com",
      "from": "Bob ",
      "subject": "Test delivered webhook"
    },
    "attachments": [

    ],
    "size": 111
  },
  "recipient": "alice@example.com",
  "event": "delivered",
  "delivery-status": {
    "tls": true,
    "mx-host": "smtp-in.example.com",
    "attempt-no": 1,
    "description": "",
    "session-seconds": 0.4331989288330078,
    "utf8": true,
    "code": 250,
    "message": "OK",
    "certificate-verified": true
  }
}

I can curl the endpoint with a specifically crafted Graylog message:

curl -XPOST https://mydomain/mailgun_logger/ -p0 -d '{"short_message":"Hello there with trailing slash in url", "host":"example.org", "facility":"test", "_foo":"bar"}'

, but the Mailgun messages (while they do show in nginx access log) are not showing in Graylog. What’s the better approach?

Thanks!
-Bronius

2

Maybe it is that GELP requires a specific format? Or at least some specific json fields? But Mailgun is shipping in on POST variables like event-data and signature… Do I sniff and rewrite in a pipeline preprocessor maybe? Will that even work for non-message-having fields?

Really it would be acceptable just to have a “ping” count: A message came in (any message) on a specific endpoint and it’s logged. I want to set up an alert for when the messages stop coming in.

Thanks!
-Bronius

3

Correct, GELF is a specific format put into JSON.

Do I sniff and rewrite in a pipeline preprocessor maybe?

Nah, pipelines etc come after receiving the message on an input. So if the GELF input rejects the message, it won’t go into any pipelines or extractors either.

I do believe you can fire up a generic HTTP receiver and tack a JSON extractor onto that. That’d do the job.

4

ok @Totally_Not_A_Robot I like the things you’re saying! . . but I expected to see this out of the box in Graylog or easily found on Graylog Marketplace, and I must be missing something. Do you know of a specific solution?

I did come across this related Feature Request closed without comment:

There is an out of the box JSON Path from HTTP API which looks like it’s a fetcher input (Graylog does the calling). I am wanting to receive pings from an external service’s webhooks and (bonus) process the inbound messages.

Thanks
-Bronius

1 Like
5

Right, I checked…

I recalled that there was a generic HTTP receiver for just about anything, but of course that doesn’t make much sense. That would’ve been something like “RAW/plaintext http”. I mean, HTTP POST <random data> doesn’t make much sense, does it? :smiley:

Of course that begs the question: does it have to be HTTP? Could you simply send in the JSON to a “RAW/plaintext TCP”?

EDIT :
Hang on… Could you not simply send the HTTP POST to the “RAW/plaintext TCP”? I don’t know whether that would work… Time to test it!

6

Ok now we’re talking! :smiley: Yes, this should at least handle pings. I’ve got to work behind an nginx proxy_pass bc of the custom port + path, but I’ll report back what I find out. Thanks!

7

Ok I think I’ve gotten in over my head proxying http to non-http TCP port listener in nginx. If someone has a solution, I’ll give it a shot, but I think next I’ll try the reverse: Have Graylog reach out to Mailgun API via JSON Path from HTTP API.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.