On-Call Tech Teams Are Fatigued and Stressed
- 1 of
-
Exhaustive Task
63% of the IT professionals surveyed said they suffer from "alert fatigue." -
Rescue Team
Alerts are taking an average of 30 minutes to solve, with three IT pros involved. -
Waste of Time
64% of the tech employees surveyed believe that up to one-quarter of all alerts are false alarms. -
Long Haul
More than one-half of on-call rotations for IT pros last for a week, and 28% last at least two weeks. -
How Users Notify On-Call Teams
Email: 82%, SMS: 57%, Telephone call: 46%, Push notification: 37%, Dashboards: 31% -
Tools Used During Issue Resolution
Chat platforms: 72%, One-on-one phone calls: 65%, Conference calls: 50%, Wiki articles: 33%, Graph tools: 30% -
How to Improve the Resolution Process
Better documentation: 56%, Enhanced reporting: 55%, Post-mortems: 50%, Greater alert accuracy: 44% -
Welcomed Solution
58% are using infrastructure automation tools. Of those using these tools, three-quarters said they make it easier to be on call. -
In-House Approach
70% of the respondents use homegrown tools or processes to solve on-call issues. -
Flexible Position
60% of the IT pros consider themselves "agile," and 52% have a year or more of DevOps experience.
On-call IT teams are responding to an average of more than 7,700 alerts from users a year, and almost a quarter of these alerts are false alarms, according to a recent survey from VictorOps. The time spent on addressing the false alarms comes out to an estimated $140,000 annual expense for organizations, findings reveal. Given the circumstances, it's easy to see why the vast majority of tech workers said they suffer from "alert fatigue." And it doesn't help that many have to pull an on-call rotation for two weeks (or longer) at a time, leading respondents to provide the following survey comments: "I get stressed, and that causes tension, which then affects my marriage." "I always have to warn my family when it's my week on call." "I have had to leave movies, cut dates short." "It affects my health due to complications of tension and anxiety over missing family events." As troubling as the situation is, tech teams that use infrastructure automation tools said the solutions help "ease their pain." More than 500 system administrators, programmers, and other IT professionals and managers took part in the research.