The Latest Cyber – Blue Screen of Death

The world came to a crashing halt the other day due to the CrowdStrike Falcon sensor software issue that caused a global outage in Microsoft Windows operating systems.

In my opinion there are two key learnings from this event for everyone.

1. Not upgrading unsupported software is not always a bad thing.

An interesting note is outlined in this recent LinkedIn post.

Southwest Airlines had no issue, while the majority of their competitors had to cancel flights, which cost them hundreds of millions of dollars.

This highlights a point I have been making for some time regarding software vendor end-of-support threats.

Operating your business on unsupported software poses no immediate threat. In most cases, unsupported software continues to run smoothly without any significant issues. However, in the longer term, the infrastructure required to run these systems eventually creates a situation where there are significant technology clashes, and ultimately, this software will not run on available servers.

You only need to look at the back end of our banking systems to know the value of old Cobol coding and AS400 servers. They continue to run the majority of the banking systems today despite the GUI front ends they put on the applications they build around them. You only have to apply for a loan or have a review on our account, and you soon see the bank staff open up a black (ironically called green) screen application.

I am confident that Microsoft would have had many detailed and robust discussions about why Southwest should migrate off the unsupported Windows 3.1. But, for whatever reason, they did not.

Every company using SAP, R3 or ECC 6 or Microsoft AX (and the list of vendors pressuring customers to upgrade goes on), will understand the pressure these vendors place on them to be on the latest (and supposedly greatest) versions of their software.

Now, I’m not suggesting you should never upgrade or update your applications—far from it. You should, but only when you have the right reasons. I am suggesting that not every unsupported system should be replaced immediately or within the deadline the software vendor puts on you.

The “why” or business case for changing systems is often no more than “Well, you will be running unsupported software.” I have clients who have changed systems purely because the risk committee of their board was uncomfortable running the business on unsupported software. And fair enough, I hear you say. It’s a risk to the company, isn’t it?

Businesses need to have a valid and compelling “Why” they should change. I have recently been involved in several projects where I didn’t feel the clients had a strong enough “why”. Wouldn’t you know it, when the time for the implementation came, they had issues, and those upgrades were as far from an overwhelming success as you can get. They were essentially an expensive screen colour and layout change. There was no initiative to improve the business or take advantage of new functionality during that project. But the risk committee was happy. So, that business incurred many millions of dollars of unnecessary expenses and added very little value to the business.

The risk committees of boards have a lot to answer for when assessing the risk of operating with unsupported software. The questions not asked seem to be:

• How many times have we had to call on the vendor’s support for issues directly related to the age of the version we are running? My guess is rarely.
• What extra business value can we leverage from this change? Do we have the appetite to make those changes?
• Do we have the internal disciplines to manage a SaaS system and all the additional demands it will place on the business?

The business case for an upgrade needs to be strong, and the appetite to apply the resources required to do the job properly needs to be high. Just like an implementation needs full-time resources dedicated to the task, so does an upgrade.

2. Regression testing of cloud-based applications, or SAAS, is now a critical discipline that every business must embed into its culture.

The cause of this major outage was a simple one. The latest code release from CrowdStrike contained a bug that had catastrophic consequences in the Windows operating environment.

Like all SAAS products, it should be tested before deployment. I have three key cybersecurity consultants within my network who advise clients to undertake regression testing.

Interestingly, in this instance, in many cases, the code was automatically applied without the chance to test it. Those companies where this patch was automatically applied saw their business come to an abrupt standstill. While those who followed good practice prevented the patch from being applied and tested, that specific release did not have the issue.

This is most likely the most important lesson of all from this event.

Inculcate regression testing into your everyday business practice. This is a business-wide challenge, not just one for the IT department.

While CrowdStrike seemingly had a fix released within a very short time, the consequences are taking so much longer to flow through and remedy because that fix requires a manual restart of all machines. It requires a human to restart a machine in “Safe Mode“ manually, remove one line of code from the package, and then restart the machine. All fixed. At least, that’s what I’ve been told.

If you are considering replacing your existing business applications, ensure you fully understand what new business disciplines will be required.

In an unrelated matter…

I was in Melbourne recently for the CeMat conference. CeMat is the major conference for warehousing and material handling technologies. I attend these conferences to stay up-to-date with what is available and working so that I can include it when necessary when devising solutions for clients.

I was having breakfast at a local cafe near my hotel when I overheard a conversation at the next table. I don’t habitually listen to those conversations, but this one piqued my interest because he mentioned the magic word – “ERP”.

This chap was explaining the project he was working on. It happened to be a “Dynamics” project, and if you believed everything he said, as his audience seemed to, it was going to cure cancer.

While I was pleased that he was enthusiastic about his work and the project, I get concerned when people take the sales pitch hook, line, and sinker. To mix metaphors, they drink way too much Cool-Aid.

This is important because while the sales pitches are correct at a high level about what an application could do, they are only sometimes correct about the specifics. And in ERP-land, specifics matter. The devil hides in the details. One of the most common comments I get from clients in post-project reviews and testimonials is, “I had no idea that …” and “I didn’t realise …” Due to my experience in this industry, my involvement adds so much value to those clients.

On a side note, I recently undertook some marketing research, and one of the most surprising pieces of feedback I received was that I was not “aggressive or vocal” enough about the value I bring. So, that was my first attempt at being more vocal about my value to your projects.

I look forward to helping you some more soon.

© David Ogilvie