If you are a regular reader of my emails and blog posts, you know that I am passionate about companies having the right financial infrastructure to operate their business. Real costs are eroding your bottom line when you don’t have a handle on people, procedures, and process.
Consider the cost of these infrastructure “fails”:
- Little to no understanding of the cost of individual services or products and whether your price covers the costs;
- The inability to seek funding from investors because you can’t pull together the required financial information;
- The cost of replacing frustrated financial staff who refuse to follow old, antiquated processes;
- Time spent by C-suite execs creating their own financial reports when their own Finance Department can’t meet their needs; and
- Fraudulent activity that goes undetected until it’s too late due to the lack of proper procedures and education.
Finance and your company’s IT capabilities are closely linked by the daily transactions that run your business. Sound, efficient infrastructure in Finance is great, but it must be supported by a highly secure and reliable IT infrastructure. I’m not speaking hypothetically, either. This reality hit home when a colleague shared with me his story of being a ransomware victim. The following reads like a script for a cybersecurity who-dun-it!
Our company uses a cloud-based server provided by Intermedia Solutions to host mission-critical applications, including our QuickBooks accounting software and our back-of-the-house order management system. The actual computer hardware on which our cloud server was running was physically located in a server farm in Atlanta, Georgia. This order management system handles everything from accepting of orders from all the channels we do business through (our own website, Amazon.com, Walmart.com, eBay and orders we take via telephone), plus it performs inventory control operations, vendor management, and purchase order issuance. Virtually everyone in the company uses one or both applications throughout every day, seven days a week. They’re accessed via Microsoft’s Remote Desktop software.
On Sunday, February 26, 2017, one of our employees logged into the server, preparing to work, and saw this message on the screen of our supposedly secure cloud server:
Whoever posted the message said that our data and applications were being held for ransom and the only way to free the data was to pay, 24 bitcoins, at the time, about $35,000. We found that the data on the server was not available to us. It has been encrypted. We were a victim of a ransomware attack.
After a moment of panic, we recalled that we and our cloud server provider had prepared for this possibility. If we hadn’t prepared, we would have been a statistic- another company who was either forced to pay the ransom or go out of business as a result of the loss of all of the company’s data. In 2017, there were 184 million ransomware attacks, most in the United States.
But we were ready and if any day was a good day for a ransomware attack, it would be a Sunday when we aren’t speaking to customers.
We had backups. Our cloud services company made image backups of the hard drive containing our cloud server and its data every night at midnight. The one thing we weren’t going to be doing was paying the ransom. Instead, we contacted Intermedia’s after-hours helpdesk and explained what happened.
We instructed them that we did not want the physical computer hardware repaired (because we didn’t now and would never trust that hardware again). Instead, we wanted a new server configured for our use. They had that ready for us in about four hours. We now had a brand-new cloud server ready to go but with none of our data on it. We then asked for a SECOND brand new cloud server to be set up for us but re-imaged from the backup image taken Saturday night at midnight. This would take longer.
Monday morning, although we were still not operating, we now had a clean, empty server and another server that APPEARED to be working with all of our applications and data on it exactly as it was at the close of business Saturday night. But I didn’t want to actually use this for fear that the ransomware application was lurking on the hard drive someplace ready to be reactivated again.
Over the next two days, we created data backups on the server and worked with our two application software companies to reinstall fresh versions of their software on the new empty server. On the third day, we did a restore of the data from the server image to the new server we planned to use. We gave instructions to Intermedia to abandon the original server that had the ransomware and the server image we had created. We were almost ready to resume operation. But I wanted to get some idea as to how we might have become victim in the first place. What I learned is that ransomware is almost always delivered via a rogue email containing an image, HTML or a PDF. The travel path for the virus was likely from one of our users who likely clicked on an email on their local computer while they were also logged into the cloud server. If that was the case, then the ransomware virus was also residing on someone’s workstation.
In my investigation, I also learned that a) Microsoft’s included anti-virus software is completely inadequate for company use and b) the ant-virus software on the server was grossly out of date.
We needed an anti-malware application that created a closed loop- coverage for the server and all of the user’s workstations that access the server. Also, it needed to be managed centrally. Users could not be trusted to keep their anti-virus software up to date. This was not the time for “free” anti-virus protection. Ultimately, I selected Symantec’s Endpoint Protection. For $28 a year per workstation/server, we got a managed malware protection suite. From a single web portal, I can see that everyone’s computers are properly protected. Then I installed it on the server and in the process, it confirmed that my restored data was clean.
Finally, on Thursday morning, we were back in full operation and properly secured.
I was pleased we had no data loss and didn’t have to pay the ransom but disappointed it took four days to recover. Here’s what I learned:
- We chose wisely when we chose Intermedia. They take our cloud-based service needs seriously.
- If you’re using computers in your business, take a good long time to think about what would happen to if you had a complete data loss, ransomware attack, etc.
- Take your IT infrastructure security needs seriously. PLAN for a worst-case breach. Do not presume that your employees keep their computer software updated.
- Don’t take your provider’s word for it that you’re protected, backups are being created, etc. Every few months I have a new server brought online and a restore performed. Once I’ve seen with my own eyes that everything works, I delete the server. It’s like conducting a fire drill.
Lessons Learned for Finance
Had Larry not had the right disaster preparedness and IT infrastructure, the costs of his crisis would have been much more than the $35,000 ransom. He still would have incurred at least 4 days of downtime. With his confidence shaken in the violated server, he still would have repeated the recovery process to bring new servers online.
Larry’s Lessons may be applicable to your own IT infrastructure, whether you’ve followed a similar process or realized that you should. Here is how Larry’s Lessons Learned can be applied to your Finance infrastructure:
- Have a disaster preparedness plan for your department that aligns with your IT disaster preparedness. Test it periodically against various scenarios, but not less than every 6 months. Update the plan based on changes in your systems, procedures or business.
- Cheaper is not always better – in fact, it rarely is. Understand your needs and invest in meeting them with the most robust tools you can afford.
- Have an IT Security Policy and related Procedures. Educate your staff at time of hire and throughout the year on the latest scams and the importance of following your company procedures.
Finally, have a third party review your processes for areas of improved efficiency and security.
Barker Associates has the unique ability to work with all sizes of organizations and building infrastructure that matters. Contact us today!
Mindy Barker, Founder & CPA | Jacksonville, FL 32256
(904) 394-2913 or (904) 728-2920 | CFO@MindyBarkerAssociates.com