Browse the comments on virtually any story about a ransomware attack and you will almost surely encounter the view that the victim organization could have avoided paying their extortionists if only they’d had proper data backups. But the ugly truth is there are many non-obvious reasons why victims end up paying even when they have done nearly everything right from a data backup perspective.

This story isn’t about what organizations do in response to cybercriminals holding their data for hostage, which has become something of a best practice among most of the top ransomware crime groups today. Rather, it’s about why victims still pay for a key needed to decrypt their systems even when they have the means to restore everything from backups on their own.

Experts say the biggest reason ransomware targets and/or their insurance providers still pay when they already have reliable backups is that nobody at the victim organization bothered to test in advance how long this data restoration process might take.

“In a lot of cases, companies do have backups, but they never actually tried to restore their network from backups before, so they have no idea how long it’s going to take,” said Fabian Wosar, chief technology officer at Emsisoft. “Suddenly the victim notices they have a couple of petabytes of data to restore over the Internet, and they realize that even with their fast connections it’s going to take three months to download all these backup files. A lot of IT teams never actually make even a back-of-the-napkin calculation of how long it would take them to restore from a data rate perspective.”

Wosar said the next most-common scenario involves victims that have off-site, encrypted backups of their data but discover that the digital key needed to decrypt their backups was stored on the same local file-sharing network that got encrypted by the ransomware.

The third most-common impediment to victim organizations being able to rely on their backups is that the ransomware purveyors manage to corrupt the backups as well.

“That is still somewhat rare,” Wosar said. “It does happen but it’s more the exception than the rule. Unfortunately, it is still quite common to end up having backups in some form and one of these three reasons prevents them from being useful.”

Bill Siegel, CEO and co-founder of Coveware, a company that negotiates ransomware payments for victims, said most companies that pay either don’t have properly configured backups, or they haven’t tested their resiliency or the ability to recover their backups against the ransomware scenario.

“It can be [that they] have 50 petabytes of backups … but it’s in a … facility 30 miles away.… And then they start [restoring over a copper wire from those remote backups] and it’s going really slow … and someone pulls out a calculator and realizes it’s going to take 69 years [to restore what they need],” Siegel told Kim Zetter, a veteran Wired reporter who recently launched a cybersecurity newsletter on Substack.

“Or there’s lots of software applications that you actually use to do a restore, and some of these applications are in your network [that got] encrypted,” Siegel continued. “So you’re like, ‘Oh great. We have backups, the data is there, but the application to actually do the restoration is encrypted.’ So there’s all these little things that can trip you up, that prevent you from doing a restore when you don’t practice.”

Wosar said all organizations need to both test their backups and develop a plan for prioritizing the restoration of critical systems needed to rebuild their network.

“In a lot of cases, companies don’t even know their various network dependencies, and so they don’t know in which order they should restore systems,” he said. “They don’t know in advance, ‘Hey if we get hit and everything goes down, these are the services and systems that are priorities for a basic network that we can build off of.'”

Wosar said it’s essential that organizations drill their breach response plans in periodic tabletop exercises, and that it is in these exercises that companies can start to refine their plans. For example, he said, if the organization has physical access to their remote backup data center, it might make more sense to develop processes for physically shipping the backups to the restoration location.

“Many victims see themselves confronted with having to rebuild their network in a way they didn’t anticipate. And that’s usually not the best time to have to come up with these sorts of plans. That’s why tabletop exercises are incredibly important. We recommend creating an entire playbook so you know what you need to do to recover from a ransomware attack.”

Microsoft today released updates to patch at least 116 security holes in its Windows operating systems and related software. At least four of the vulnerabilities addressed today are under active attack, according to Microsoft.

Thirteen of the security bugs quashed in this month’s release earned Microsoft’s most-dire “critical” rating, meaning they can be exploited by malware or miscreants to seize remote control over a vulnerable system without any help from users.

Another 103 of the security holes patched this month were flagged as “important,” which Microsoft assigns to vulnerabilities “whose exploitation could result in compromise of the confidentiality, integrity, or availability of user data, or of the integrity or availability of processing resources.”

Among the critical bugs is of course the official fix for the PrintNightmare print spooler flaw in most versions of Windows (CVE-2021-34527) that prompted Microsoft to rush out a patch for a week ago in response to exploit code for the flaw that got accidentally published online. That patch seems to have caused a number of problems for Windows users. Here’s hoping the updated fix resolves some of those issues for readers who’ve been holding out.

CVE-2021-34448 is a critical remote code execution vulnerability in the scripting engine built into every supported version of Windows — including server versions. Microsoft says this flaw is being exploited in the wild.

Both CVE-2021-33771 and CVE-2021-31979 are elevation of privilege flaws in the Windows kernel. Both are seeing active exploitation, according to Microsoft.

Chad McNaughton, technical community manager at Automox, called attention to CVE-2021-34458, a remote code execution flaw in the deepest areas of the operating system. McNaughton said this vulnerability is likely to be exploited because it is a “low-complexity vulnerability requiring low privileges and no user interaction.”

Another concerning critical vulnerability in the July batch is CVE-2021-34494, a dangerous bug in the Windows DNS Server that earned a CVSS score (severity) of 9.8 out of a possible 10.

“Both core and full installations are affected back to Windows Server 2008, including versions 2004 and 20H2,” said Aleks Haugom, also with Automox.

“DNS is used to translate IP addresses to more human-friendly names, so you don’t have to remember the jumble of numbers that represents your favorite social media site,” Haugom said. “In a Windows Domain environment, Windows DNS Server is critical to business operations and often installed on the domain controller. This vulnerability could be particularly dangerous if not patched promptly.”

Microsoft also patched six vulnerabilities in Exchange Server, an email product that has been under siege all year from attackers. Satnam Narang, staff research engineer at Tenable, noted that while Microsoft says two of the Exchange bugs tackled this month (CVE-2021-34473 and CVE-2021-34523) were addressed as part of its security updates from April 2021, both CVEs were somehow omitted from that April release. Translation: If you already applied the bevy of Exchange updates Microsoft made available in April, your Exchange systems have protection against these flaws.

Other products that got patches today include Microsoft Office, Bing, SharePoint Server, Internet Explorer, and Visual Studio. The SANS Internet Storm Center as always has a nice visual breakdown of all the patches by severity.

Adobe also issued security updates today for Adobe Acrobat and Reader, as well as Dimension, Illustrator, Framemaker and Adobe Bridge.

Chrome and Firefox also recently have shipped important security updates, so if you haven’t done so recently take a moment to save your tabs/work, completely close out and restart the browser, which should apply any pending updates.

The usual disclaimer:

Before you update with this month’s patch batch, please make sure you have backed up your system and/or important files. It’s not uncommon for Windows updates to hose one’s system or prevent it from booting properly, and some updates even have been known to erase or corrupt files.

So do yourself a favor and backup before installing any patches. Windows 10 even has some built-in tools to help you do that, either on a per-file/folder basis or by making a complete and bootable copy of your hard drive all at once.

And if you wish to ensure Windows has been set to pause updating so you can back up your files and/or system before the operating system decides to reboot and install patches on its own schedule, see this guide.

As always, if you experience glitches or problems installing any of these patches this month, please consider leaving a comment about it below; there’s a better-than-even chance other readers have experienced the same and may chime in here with some helpful tips. Also, check out AskWoody, which keeps a close eye out for specific patches that may be causing problems for users.

Last summer, financial institutions throughout Texas started reporting a sudden increase in attacks involving well-orchestrated teams that would show up at night, use stolen trucks and heavy chains to rip Automated Teller Machines (ATMs) out of their foundations, and make off with the cash boxes inside. Now it appears the crime — known variously as “ATM smash-and-grab” or “chain gang” attacks — is rapidly increasing in other states.

The Texas Bankers Association documented at least 139 chain gang attacks against Texas financial institutions in the year ending November 2020. The association says organized crime is the main source of the destructive activity, and that Houston-based FBI officials have made more than 50 arrests and are actively tracking about 250 individuals suspected of being part of these criminal rings.

From surveillance camera footage examined by fraud investigators, the perpetrators have followed the same playbook in each incident. The bad guys show up in the early morning hours with a truck or tractor that’s been stolen from a local construction site.

Then two or three masked men will pry the front covering from the ATM using crowbars, and attach heavy chains to the cash machine. The canisters of cash inside are exposed once the crooks pull the ATM’s safe door off using the stolen vehicle.

In nearly all cases, the perpetrators are done in less than five minutes.

Tracey Santor is the bond product manager for Travelers, which insures a large number of financial institutions against this type of crime. Santor said investigators questioning some of the suspects learned that the smash-and-grabs are used as a kind of initiation for would-be gang members.

“One of the things they found out during the arrest was the people wanting to be in the gang were told they had to bring them $250,000 within a week,” Santor said. “And they were given instructions on how to do it. I’ve also heard of cases where the perpetrators put construction cones around the ATM so it looks to anyone passing by that they’re legitimately doing construction at the site.”

Santor said the chain gang attacks have spread to other states, and that in the year ending June 2021 Travelers saw a 257 percent increase in the number of insurance claims related to ATM smash-and-grabs.

That 257 percent increase also includes claims involving incidents where attackers will crash a stolen car into a convenience store, and then in the ensuing commotion load the store’s ATM into the back of the vehicle and drive away.

In addition to any cash losses — which can often exceed $200,000 — replacing destroyed ATMs and any associated housing can take weeks, and newer model ATMs can cost $80,000 or more.

“It’s not stopping,” Santor said of the chain gang attacks. “In the last year we counted 32 separate states we’ve seen this type of attack in. Normally we are seeing single digits across the country. 2021 is going to be the same or worse for us than last year.”

Increased law enforcement scrutiny of the crime in Texas might explain why a number of neighboring states are seeing a recent uptick in the number of chain gang attacks, said Elaine Dodd, executive vice president of the fraud division for the Oklahoma Bankers Association.

“We have a lot of it going on here now and they’re getting good at it,” Dodd said. “The numbers are surging. I think since Texas has focused law enforcement attention on this it’s spreading like fingers out from there.”

It’s not hard to see why physical attacks against ATMs are on the rise. In 2019, the average amount stolen in a traditional bank robbery was just $1,797, according to the FBI.

In contrast, robbing ATMs is way less risky and potentially far more rewarding for the perpetrators. That’s because bank ATMs can typically hold hundreds of thousands of dollars in cash.

Dodd said she hopes to see more involvement from federal investigators in fighting chain gang attacks, and that it would help if more of these attacks were prosecuted as bank robberies, which can carry stiff federal penalties. As it is, she said, most incidents are treated as property crimes and left to local investigators.

“We had a rash of three attacks recently and contacted the FBI, and were told, ‘We don’t work these,'” Dodd said. “The FBI looks at these attacks not as bank robbery, but just the theft of cash.”

In January, Texas lawmakers are introduced legislation that would make destroying an ATM a third degree felony offense. Such a change would mean chain gang members could be prosecuted with the same zeal Texas applies to people who steal someone’s livestock, a crime which is punishable by 2-10 years in prison and a fine of up to $10,000 (or both).

“The bottom line is, right now bank robbery is a felony and robbing an unattended ATM is not,” Santor said.

KrebsOnSecurity checked in with the European ATM Security Team (EAST), which maintains statistics about fraud of all kinds targeting ATM operators in Europe. EAST Executive Director Lachlan Gunn said overall physical attacks on ATMs in Europe have been a lot quieter since the pandemic started.

“Attacks fell right away during the lockdowns and have started to pick up a little as the restrictions are eased,” Gunn said. “So no major spike here, although [the United States is] further ahead when it comes to the easing of restrictions.”

Gunn said the most common physical attacks on European ATMs continue to involve explosives —  such as gas tanks and solid explosives that are typically stolen from mining and construction sites.

“The biggest physical attack issue in Europe remains solid explosive attacks, due to the extensive collateral damage and the risk to life,” Gunn said.

The Texas Bankers Association report, available here (PDF), includes a number of recommended steps financial institutions can take to reduce the likelihood of being targeted by chain gangs.

Last week cybercriminals deployed ransomware to 1,500 organizations that provide IT security and technical support to many other companies. The attackers exploited a vulnerability in software from Kaseya, a Miami-based company whose products help system administrators manage large networks remotely. Now it appears Kaseya’s customer service portal was left vulnerable until last week to a data-leaking security flaw that was first identified in the same software six years ago.

On July 3, the REvil ransomware affiliate program began using a zero-day security hole (CVE-2021-30116) to deploy ransomware to hundreds of IT management companies running Kaseya’s remote management software — known as the Kaseya Virtual System Administrator (VSA).

According to this entry for CVE-2021-30116, the security flaw that powers that Kaseya VSA zero-day was assigned a vulnerability number on April 2, 2021, indicating Kaseya had roughly three months to address the bug before it was exploited in the wild.

Also on July 3, security incident response firm Mandiant notified Kaseya that their billing and customer support site —portal.kaseya.net — was vulnerable to CVE-2015-2862, a “directory traversal” vulnerability in Kaseya VSA that allows remote users to read any files on the server using nothing more than a Web browser.

As its name suggests, CVE-2015-2862 was issued in July 2015. Six years later, Kaseya’s customer portal was still exposed to the data-leaking weakness.

Mandiant notified Kaseya after hearing about it from Alex Holden, founder and chief technology officer of Milwaukee-based cyber intelligence firm Hold Security. Holden said the 2015 vulnerability was present on Kaseya’s customer portal until Friday afternoon, allowing him to download the site’s “web.config” file, a server component that often contains sensitive information such as usernames and passwords and the locations of key databases.

“It’s not like they forgot to patch something that Microsoft fixed years ago,” Holden said. “It’s a patch for their own software. And it’s not zero-day. It’s from 2015!”

The official description of CVE-2015-2862 says a would-be attacker would need to be already authenticated to the server for the exploit to work. But Holden said that was not the case with the vulnerability on the Kaseya portal that he reported via Mandiant.

“This is worse because the CVE calls for an authenticated user,” Holden said. “This was not.”

Michael Sanders, executive vice president of account management at Kaseya, confirmed that the customer portal was taken offline Friday in response to a vulnerability report. Sanders said the portal had been retired in 2018 in favor of a more modern customer support and ticketing system, yet somehow the old site was still left available online.

“It was deprecated but left up,” Sanders said.

In a written statement shared with KrebsOnSecurity, Kaseya said that in 2015 CERT reported two vulnerabilities in its VSA product.

“We worked with CERT on responsible disclosure and released patches for VSA versions V7, R8, R9 and R9 along with the public disclosure (CVEs) and notifications to our customers. Portal.kaseya.net was not considered by our team to be part of the VSA shipping product and was not part of the VSA product patch in 2015. It has no access to customer endpoints and has been shut down – and will no longer be enabled or used by Kaseya.”

“At this time, there is no evidence this portal was involved in the VSA product security incident,” the statement continued. “We are continuing to do forensic analysis on the system and investigating what data is actually there.”

The REvil ransomware group said affected organizations could negotiate independently with them for a decryption key, or someone could pay $70 million worth of virtual currency to buy a key that works to decrypt all systems compromised in this attack.

But Sanders said every ransomware expert Kaseya consulted so far has advised against negotiating for one ransom to unlock all victims.

“The problem is that they don’t have our data, they have our customers’ data,” Sanders said. “We’ve been counseled not to do that by every ransomware negotiating company we’ve dealt with. They said with the amount of individual machines hacked and ransomwared, it would be very difficult for all of these systems to be remediated at once.”

In a video posted to Youtube on July 6, Kaseya CEO Fred Voccola said the ransomware attack had “limited impact, with only approximately 50 of the more than 35,000 Kaseya customers being breached.”

“While each and every customer impacted is one too many, the impact of this highly sophisticated attack has proven to be, thankfully, greatly overstated,” Voccola said.

The zero-day vulnerability that led to Kaseya customers (and customers of those customers) getting ransomed was discovered and reported to Kaseya by Wietse Boonstra, a researcher with the Dutch Institute for Vulnerability Disclosure (DIVD).

In a July 4 blog post, DIVD’s Victor Gevers wrote that Kaseya was “very cooperative,” and “asked the right questions.”

“Also, partial patches were shared with us to validate their effectiveness,” Gevers wrote. “During the entire process, Kaseya has shown that they were willing to put in the maximum effort and initiative into this case both to get this issue fixed and their customers patched. They showed a genuine commitment to do the right thing. Unfortunately, we were beaten by REvil in the final sprint, as they could exploit the vulnerabilities before customers could even patch.”

Still, Kaseya has yet to issue an official patch for the flaw Boonstra reported in April. Kaseya told customers on July 7 that it was working “through the night” to push out an update.

Gevers said the Kaseya vulnerability was discovered as part of a larger DIVD effort to look for serious flaws in a wide array of remote network management tools.

“We are focusing on these types of products because we spotted a trend where more and more of the products that are used to keep networks safe and secure are showing structural weaknesses,” he wrote.

Microsoft on Tuesday issued an emergency software update to quash a security bug that’s been dubbed “PrintNightmare,” a critical vulnerability in all supported versions of Windows that is actively being exploited. The fix comes a week ahead of Microsoft’s normal monthly Patch Tuesday release, and follows the publishing of exploit code showing would-be attackers how to leverage the flaw to break into Windows computers.

At issue is CVE-2021-34527, which involves a flaw in the Windows Print Spooler service that could be exploited by attackers to run code of their choice on a target’s system. Microsoft says it has already detected active exploitation of the vulnerability.

Satnam Narang, staff research engineer at Tenable, said Microsoft’s patch warrants urgent attention because of the vulnerability’s ubiquity across organizations and the prospect that attackers could exploit this flaw in order to take over a Windows domain controller.

“We expect it will only be a matter of time before it is more broadly incorporated into attacker toolkits,” Narang said. “PrintNightmare will remain a valuable exploit for cybercriminals as long as there are unpatched systems out there, and as we know, unpatched vulnerabilities have a long shelf life for attackers.”

In a blog post, Microsoft’s Security Response Center said it was delayed in developing fixes for the vulnerability in Windows Server 2016, Windows 10 version 1607, and Windows Server 2012. The fix also apparently includes a new feature that allows Windows administrators to implement stronger restrictions on the installation of printer software.

“Prior to installing the July 6, 2021, and newer Windows Updates containing protections for CVE-2021-34527, the printer operators’ security group could install both signed and unsigned printer drivers on a printer server,” reads Microsoft’s support advisory. “After installing such updates, delegated admin groups like printer operators can only install signed printer drivers. Administrator credentials will be required to install unsigned printer drivers on a printer server going forward.”

Windows 10 users can check for the patch by opening Windows Update. Chances are, it will show what’s pictured in the screenshot below — that KB5004945 is available for download and install. A reboot will be required after installation.

Friendly reminder: It’s always a good idea to backup your data before applying security updates. Windows 10 has some built-in tools to help you do that, either on a per-file/folder basis or by making a complete and bootable copy of your hard drive all at once.

This post will be updated if Windows users start reporting any issues in applying the patch.