Fire drills help align your incident response team and prepare them for real incidents. We recommend running a fire drill to get familiar with the FireHydrant platform, but we also recommend running fire drills often (once per month or once per quarter).
Tips for success
- Create a low-stress environment for the fire drill.
- Focus on the incident management process and workflows.
- Encourage participation!
Here are 6 tips for running a fire drill in FireHydrant.
1. Put a plan in place
For this first fire drill, it's probably best to have a plan in place and to put it on the calendar. This fire drill is really an opportunity for your team to get more familiar with the FireHydrant platform. For future fire drills, you may consider a more unannounced approach.
Make sure your FireHydrant runbook(s) are good to go. Review them to ensure all of the steps you want are in place but don't worry about perfection. Chances are you'll learn things from your first fire drill and make some changes to your runbooks.
2. Create a scenario
We recommend taking a recent incident your team worked through and recreating that scenario for your fire drill. Set the stage briefly so you have a situation to base your fire drill on.
3. Declare an incident
Now that the stage is set, declare an incident in FireHydrant. There are a number of ways you can do this depending on how you've configured things. The simplest way would be to enter
/firehydrant new into Slack. Use the "game day" severity to signify this incident is a fire drill.
Tip: We support the following aliases for all slash commands:
As you work through the incident, star important messages in Slack so that they're automatically captured in the timeline and the retrospective.
Here are some common Slack commands you'll find useful during an incident:
Add a note
/firehydrant add note [note text]
Think of notes as a piece of information more important than a normal chat message. You are able to filter specifically for notes on the FireHydrant incident timeline.
Set the impact
/firehydrant add impact
This will open a text box that you can type in to start filtering through all available services, environments, and functionalities in your application.
Update the incident
The update command will allow you to change the milestone, component status conditions, and include a comment as to why you are changing the status. If you have active status pages you can choose to post your update to the status pages as well.
Assign a role
/firehydrant assign role
Assigning a role to someone that has linked their Slack account will also send that user a DM on Slack informing them that they have been assigned a role to an active incident.
Insert artifacts (graphs, screenshots, etc.)
You can star any messages or images in Slack and they'll appear in your retrospective.
4. Resolve the incident
Once the incident is deemed "resolved", resolve it in FireHydrant. The easiest way to do that is by entering
/firehydrant resolve into the Slack channel. This will close an incident and set its status to resolved. A resolved incident will continue to collect chat messages.
5. Run a retrospective
Now that the incident is resolved, it's time to run a retrospective! For the most part, you can run your incidents entirely out of Slack, however, retrospectives need to be done inside the FireHydrant UI. Learn more about running retrospectives in FireHydrant here.
Tip: you can re-order and modify the retrospective questions.
6. Review your analytics
Once you've completed the retrospective, head over to the analytics section of the FireHydrant UI to see what types of metrics we track. Learn more about analytics in FireHydrant here.