Hi,
I’m trying to connect Graylog server with PagerDuty alerting, however it doesn’t seem to work for unknown reason. Notification settings say alert was successfully sent but PagerDuty never shows any alert or test notification.
Server log doesn’t show any new entries when attempting to send out test alert, logs show there’s 202 ACCEPTED response, API keys are correct. The server has no problems with sending alerts to Slack or e-mail. I’ve tried using both Events V2 and Graylog integration service.
Is there anything that I could still do?
Environment:
OS: Ubuntu 20.04
Graylog Version: 4.1.14-1
Log response:
2022-06-20T15:50:00.329+02:00 DEBUG [PagerDutyClient] Triggering event in PagerDuty with POST payload: {“routing_key”:“EXAMPLE”,“event_action”:“trigger”,“dedup_key”:“Graylog//Event Definition Test Title”,“client”:“Graylog/”,“client_url”:“http://graylog.example.com”,“payload”:{“summary”:“Notification test message triggered from user local:admin”,“severity”:“warning”,“component”:“GraylogAlerts”,“source”:“Graylog:”,“class”:“alerts”,“timestamp”:“2022-06-20T13:50:00.327Z”,“group”:""}}
2022-06-20T15:50:01.206+02:00 DEBUG [PagerDutyClient] PagerDuty POST completed in 191ms [HTTP 202]. Response body: {“status”:“success”,“message”:“Event processed”,“dedup_key”:“Graylog//Event Definition Test Title”}
If I understand this correct, there are no visual issues with Graylog sending message/s but the message/s alert is not being received on PagerDuty?
If this is correct can you versify my tcpdump or Wireshark that the data packet is showing up on PagerDuty?
It looks like PagerDuty gets the test alert but doesn’t know what to do with it later - Graylog notification settings say " Success: Notification was executed successfully.". There’s multiple packets getting received by filter with commands “tcpdump src pagerduty.com” and “tcpdump dst pagerduty.com”. As the debug logs show above, I get 202 response from PagerDuty when test alert is sent.
This is odd, unfortunately, I have not worked with PagerDuty. What I have seen with this type of issue was firewall/Iptables or SELinux/Apparmor causing issues. From what you showed it should be working. I assume you look at all the other log files on this system? Double check you configurations.
I just looked into PagerDuty here and here? Was this how you installed it? The reason I ask this I wanted to lab this out.
Hello,
There is no egress limit on firewall, I haven’t really found anything interesting in other logs. Configs seem to be fine.
I haven’t used any of those plugins from the links. I used Graylog 4.1 PagerDuty plugin that’s in the main code now. I upgraded from version 3.1 to 4.1 specifically to use it, there never was any PagerDuty plugin installed before.
I’m not sure if there are any weird tweaks, I’m not the one that installed it in the first place. I tried to look into bash history and I didn’t find anythign strange but I’m not really familiar with installation process.
Oh boy, I hate that when I get someone else’s work and they didn’t document it. I feel your frustration.
Not much more I can do to help, maybe someone lese here has installed or configured PagerDuty.
