PagerDuty

Our PagerDuty integration enables various functions to improve the efficiency of your incident management. You can:

  • Automatically create FireHydrant incidents from PagerDuty alerts
  • Notify different channels based on incoming alert content
  • Page out to teams and service in PagerDuty from FireHydrant
  • Automatically pull in on-call responders from PagerDuty's escalation policies to incident channels
  • Importing services from PagerDuty to FireHydrant's Service Catalog
  • ...and more!

Prerequisites

  • You will need Owner permissions to configure integrations in FireHydrant
  • You'll need access to PagerDuty's Webhooks and Integrations settings (usually admin).

Installation

First, you need a PagerDuty API key.

  1. Go to your PagerDuty dashboard and navigate to the page where you can set up API keys. (This will be something like Integrations > API Access Keys ).
API Access Keys menu item in PagerDuty

API Access Keys menu item in PagerDuty

  1. On the API Access page, click Create New API Key and provide a description for the API key.
  2. Click Create Key.
  3. Copy the API Key that PagerDuty generates. Then go to FireHydrant's integrations page.
  4. Click on the PagerDuty integration card, then click Setup PagerDuty.
  5. Enter the PagerDuty API key you copied into the API Token field.
  6. For Domain, we default to showing https://api.pagerduty.com, which is the domain for PagerDuty's US service region. If you are a company based out of the EU and using PagerDuty's EU service region, then you will need to modify that value to https://api.eu.pagerduty.com. For more information, visit PagerDuty's documentation.
  7. Click Authorize Application.

Setup Outgoing Webhook

🚧

Note

If you plan to import or link services from PagerDuty, do not configure this outgoing webhook and skip this section. When you link or import PagerDuty Services to FireHydrant services, we automatically create service-scoped webhooks in PagerDuty.

After configuring the connection, if you do not plan to import or link services from PagerDuty, you should configure an outgoing webhook from PagerDuty so FireHydrant can both sync incident states and receive webhooks on alerts.

Webhook URL generated after installing PagerDuty

Webhook URL generated after installing PagerDuty

  1. In the integration settings page for PagerDuty on FireHydrant, copy the webhook address provided.
  2. In PagerDuty, Head to Integrations > Generic Webhooks and click "+ New Webhook."
  3. In Webhook URL, paste in the value you copied from FireHydrant.
  4. For Scope Type choose "Account".
  5. For Event Subscription, you can uncheck all the service.* events, leaving only the incident updates.
  6. Click "Add Webhook".

To test this, go ahead and create an incident in FireHydrant. With default rules in FireHydrant, this will log an alert processing message.

Webhook logged after PagerDuty alert created

Webhook logged after PagerDuty alert created

PagerDuty Alert Routing

Configuring alert routes in PagerDuty

Configuring alert routes in PagerDuty

Once your PagerDuty instance is configured, you can set up Alert Routes to take action on your alerts based on the data included in the alert. You can automatically open new incidents, send alerts to any Slack channel, log an alert in FireHydrant, or simply ignore it. To learn more, visit Alert Routing documentation. The rest of this page details the parameter mappings for PagerDuty.

Parameter Mappings

Here is the table of routable parameters on FireHydrant and the corresponding key/value from the inbound PagerDuty webhook(s). The $ refers to the webhook body content as a JSON object.

An explanation of PagerDuty's Webhook content can be found in their docs here.

Parameter NamePagerDuty Webhook BodyNotes
PagerDuty: Event Type$.event.event_typeThe type of event sent by PagerDuty
PagerDuty: Occurred At$.event.occurred_atThe datetime of when the event occurred according to PagerDuty
PagerDuty: Alert ID$.event.data.idThe ID of the alert in PagerDuty
PagerDuty: Incident ID$.event.data.incident.idThe ID of the incident in PagerDuty
PagerDuty: Incident Summary$.event.data.titleShort summary of the PagerDuty incident
PagerDuty: Data Type$.event.data.typeThe type of data, e.g., 'incident' or 'incident_note', present in this message
PagerDuty: Alert Web URL$.event.data.html_urlURL to alert page in PagerDuty
PagerDuty: Incident Web URL$.event.data.incident.html_urlURL to incident page in PagerDuty
PagerDuty: Incident Priority$.event.data.priority.summaryIncident priority in PagerDuty. Must enable priorities in PagerDuty to use this.
PagerDuty: Incident Status$.event.data.statusThe status of the incident in PagerDuty. Is one of triggered, acknowledged, and resolved.
PagerDuty: Incident Note Content$.event.data.contentThe content of a note posted to an incident in PagerDuty. Only shows up for incident.annotated events.
PagerDuty: Incident Title$.event.data.titleName of the incident. Essentially the same as Incident Summary.

The following table maps our overall Alert Routing mapping object - these parameters are standard across all Alerting/Monitoring integrations.

Parameter NamePagerDuty Webhook BodyNotes
Alert Summary$.event.data.titleThe same as PagerDuty: Incident Summary above.
Alert DescriptionN/AMaps to nothing for PagerDuty. PagerDuty has a Description field in their alerts, but oddly they do not surface it in the webhook, so FireHydrant has no way of accessing it.
Alert Priority$.event.data.priority.summaryThe same as PagerDuty: Incident Priority above.
Alert Status$.event.data.statusAlmost the same as PagerDuty: Incident Status above. However, FireHydrant expects opened instead of triggered to align with internal terminology.
Alert Associated InfrastructureN/AThe associated Service/Functionality in FireHydrant. For us to detect this, you need to import and/or link a service in PagerDuty to FireHydrant.

Next Steps

Now that you've configured the Opsgenie integration, you can take the following steps to get the full power of FireHydrant + Opsgenie: