In a previous blog post, we explained how careless IT development leads to time loss — and financial loss. But it’s not only avoidable rework that unnecessarily increases your IT infrastructure costs. IT systems can also simply fail, causing another major financial impact. If employees can’t do their job or customers can’t place orders, the problems spiral from there. But how do you avoid the risk of downtime? And how do you keep that downtime as short as possible, if disaster strikes despite your best efforts?
Every second that you can’t pick orders in your warehouse due to an IT failure is a second of lost profit. Every second that your customers can’t order or pay on your website is costing your money. These are just two examples of how a disrupted IT infrastructure can inflict major financial damage due to lost time — sometimes literally by the second.
Even if it’s not absolutely clear at first glance how much IT downtime costs your organization, you should probably think about it for a moment. Do the calculation for yourself, maybe with the help of an expert, and you’ll be shocked. The point is to minimize the risk of IT downtime. And, if Murphy’s law does strike, to reduce the mean time to recovery as much as possible. But how do you do that?
First prevent, then cure
You start with something we’ve already mentioned in our blog about the cost of rework. If you reduce expenditure on rework and invest the savings in IT automation, you immediately improve the quality of your IT environment, reduce the chance of errors, and slash the risk of expensive downtime.
Even then, you run the risk that your IT systems will fail at some point. So, if you can shorten the time needed to restore, you’ll at least succeed in significantly limiting the damage. You can’t do much about the per-second financial damage you suffer due to downtime, but there’s certainly a lot you can do to minimize the number of seconds that the damage takes place.
Prepare for the worst
Ideally, you should prepare yourself according to the chaos engineering theory, an idea that comes from Netflix. What would happen if you let a wild monkey loose in your data center and it was capable of doing anything at any time, such as pulling out a cable or switching off a server? Could you solve every potential problem using your existing IT architecture? The point is that you should build as much resilience and resistance as possible into your IT environment.
To do this, you need to gain clear insight into what’s going wrong as quickly as possible. You do this via an observability platform, but also much earlier by providing suitable solutions for possible issues when designing your IT architecture. These solutions should prevent downtime, for example by focusing on high availability, or by allowing the system to repair itself. And if the system does grind to a halt, you should have solutions for rapid disaster recovery, based on infrastructure-as-code, automated configuration, and automatically created images, etc.
Balancing risk and ROI
These are all elements that you might not always think about when building your infrastructure. That isn’t necessarily a problem, as long as you have an experienced partner you can count on, one who can help you come up with a solution designed around the level of risk tolerance you’re comfortable with. Do you want a solution that kicks in instantly every time the proverbial monkey makes an appearance, with zero downtime as the goal? Or can you live with a certain degree of risk?
At BRYXX, we specialize in crafting these solutions — and we calculate exactly how much a DevOps transformation will cost you, by how much it will increase your profit (by reducing downtime recovery time), and how long it will take to earn back your investment. The truth is that with a modest investment, you can quickly reduce your downtime and you’ll be saving money in no time. And you’ll be greatly reducing the risk of catastrophic losses. In this way, you turn the investment in IT from a cost item into a profit.