Building resilience and boosting observability for a telecoms giant

Major outages to sales and service channels were too frequent, meaning customers couldn’t pay bills or make changes to their accounts, and often at key transaction times. Not only were these outages frustrating because the client was unable to identify the cause of the problems quickly, but some of these were very public and damaging to the brand itself. With such a huge volume of coverage, it’s business critical that disruptions like this are managed quickly and effectively.

The client needed an experienced partner with enterprise software expertise to review how they monitor, respond to and prevent major incidents with their core online channels (collectively, their Site Reliability Engineering - SRE - practices), give them a clear view of what was happening, and get them to a place of action. 

First, we needed an overview of the business structure and operations. An initial diagnostic phase gave us a holistic understanding of the system issues and where the problems lay, meaning we could objectively assess not just the technology, but the resources and processes the client had in place to tackle the service downtime.

Using our proprietary Accelerate framework, we identified core issues with monitoring and alerting, incident management and observability. The diagnostic phase revealed an over-reliance on manual operations, and an inability to effectively prioritise tasks, making the team reactive instead of proactive.