The financial fallout of getting IT change wrong

The tale of financial damage from IT change going wrong has existed as long as business computers and their networks have. But in today’s climate, it’s never been stronger or riskier.

IT infrastructure is larger and more complex, and while systems are more agile, change has to take place faster and is harder to oversee.

And the greatest financial risk comes from rushed change.

Research has revealed that the average cost of unplanned downtime is over $14,000 per minute, with this figure rising to nearly $24,000 for large enterprises.

Of course, these costs vary depending on factors like the sector and the company’s internal environment. But the root causes of unplanned downtime are all too familiar.

The reality is most outages and audit failures come from unmanaged change – this could be a routine system upgrade before a support period ends or a configuration issue that has gone unnoticed. But modernization should be driven by ongoing control and visibility of systems, not by upcoming deadlines.

So, what is the cost of unmanaged change and an IT incident? And how can companies make risk visible?

The costs of untracked and rushed change

Unplanned downtime can lead to companies racking up some high immediate costs. Financial services firms can see between $5-7 million in costs per hour, as trading halts and transactions fail. Retailers face lost sales and cart abandonment, while manufacturing companies can suffer from production stoppage and supply chain disruption.

Yet it’s not only the immediate financial damage organizations have to consider: these incidents trigger reputational damage and operational waste, as well as the potential for regulatory fines.

But really, these costs all come from untracked and rushed change, be it to deal with a looming end-of-support deadline or to respond to an incident of unmanaged change when staff have implemented ‘fixes’ without following the proper change process.

The problem is when change is underscored by urgency rather than strategy, companies can end up paying significantly more than if that change had been controlled and visible – rushed change can lead to expensive upgrades in order to “get it done” or to rectify unplanned downtime.

What’s more, as change is rushed, the necessary processes and tooling might not be in place to predict or mitigate downtime. Nearly two in five organizations, for example, have experienced major outages caused by human error, most of them rooted in procedure failure.

And as we’ve seen, unplanned outages cost thousands per minute – around 35% more than planned downtime – demonstrating why continuous change visibility is so important.

The total cost distribution of incidents

The total cost from an incident can be expansive, both directly and indirectly, and over the short and longer term. Immediately, there is lost revenue, in terms of sales and abandoned transactions. In the longer term, the incident can cause contractual breaches around Service Level Agreements (SLAs) that are in place around services provided.

An incident such as an outage will breach the SLA and mean that penalties, known as customer or service credits, will have to be paid to any clients dependent on the impact on performance.

Then there is the labor required to carry out the immediate incident response, the war room hours spent understanding the suspected root causes of the event and how to resolve it. Investigations into failures can subsequently go on for a long time, requiring software tools to detect the causes behind the incident, consultants – especially if it involves outdated IT infrastructure where more specialist knowledge is required – and anything from post-mortem, manual remediation work to audit and legal activities, all adding to overall recovery costs.

Why it’s not innovation, but risk, that drives spend

The financial fallout from unplanned downtime illustrates how risk is the ultimate driver of spend, even if there can be the perception that innovation consumes a large chunk of IT budgets.

But rather than overspending on modernization projects in a rush to meet deadlines or to exercise damage limitation as downtime takes place – with resources sucked up in time-consuming remediation processes – organizations should look to invest in change visibility and modernization on an ongoing basis.

Many online analyses routinely attribute the majority of major incidents to failed, unauthorized or poorly controlled changes rather than infrastructure faults. Nearly every firewall breach (99%), for example, is caused by misconfiguration, according to Gartner research.

And a lack of change visibility and auditability contributes to longer outages and higher incident frequency. So, companies need to make risk visible, and that will empower them to improve their mean time to detect outages and also their mean time to recover from them.

Making risk visible

But what does visibility look like? Visibility means being able to detect and validate any changes taking place in an organization's IT estate, both on the cloud and on-premise, in real time. This depends on automatically and continuously extracting data from these sources and presenting this information on a central platform to provide a full picture of changes happening or about to happen.

A key part of this process centers on establishing a configuration baseline – what is the minimum acceptable level that all systems and applications have to adhere to (e.g. what operating system version are they running on, what are their registry settings)? IT teams can then detect any changes, see whether they cause an application to drift away from this baseline and take action.

This isn’t to say change shouldn’t happen – it’s crucial for maintaining the security and hygiene of IT estates. But when it’s forced or invisible, it becomes a primary driver of risk and severe financial fallout. That’s why it’s integral organizations invest in ensuring change is visible, managed and controlled.

We list the best Linux distros for Windows users.

This article was produced as part of TechRadar Pro Perspectives, our channel to feature the best and brightest minds in the technology industry today.

The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/pro/perspectives-how-to-submit