Roughly one year ago, KrebsOnSecurity published a lengthy investigation into the individuals behind Coinhive[.]com, a cryptocurrency mining service that has been heavily abused to force hacked Web sites to mine virtual currency. On Tuesday, Coinhive announced plans to pull the plug on the project early next month.

In March 2018, Coinhive was listed by many security firms as the top malicious threat to Internet users, thanks to the tendency for Coinhive’s computer code to be surreptitiously deployed on hacked Web sites to steal the computer processing power of its visitors’ devices.

Coinhive took a whopping 30 percent of the cut of all Monero currency mined by its code, and this presented something of a conflict of interest when it came to stopping the rampant abuse of its platform. At the time, Coinhive was only responding to abuse reports when contacted by a hacked site’s owner. Moreover, when it would respond, it did so by invalidating the cryptographic key tied to the abuse.

Trouble was, killing the key did nothing to stop Coinhive’s code from continuing to mine Monero on a hacked site. Once a key was invalidated, Coinhive would simply cut out the middleman and proceed to keep 100 percent of the cryptocurrency mined by sites tied to that account from then on.

In response to that investigation, Coinhive made structural changes to its platform to ensure it was no longer profiting from this shady practice.

Troy Mursch is chief research officer at Bad Packets LLC, a company that has closely chronicled a number of high-profile Web sites that were hacked and seeded with Coinhive mining code over the years. Mursch said that after those changes by Coinhive, the mining service became far less attractive to cybercriminals.

“After that, it was not exactly enticing for miscreants to use their platform,” Mursch said. “Most of those guys just took their business elsewhere to other mining pools that don’t charge anywhere near such high fees.”

As Coinhive noted in the statement about its closure, a severe and widespread drop in the value of most major crytpocurrencies weighed heavily on its decision. At the time of my March 2018 piece on Coinhive, Monero was trading at an all-time high of USD $342 per coin, according to charts maintained by coinmarketcap.com. Today, a single Monero is worth less than $50.

In the announcement about its pending closure, Coinhive said the mining service would cease to operate on March 8, 2019, but that users would still be able to access their earnings dashboards until the end of April. However, Coinhive noted that only those users who had earned above the company’s minimum payout threshold would be able to cash out their earnings.

Mursch said it is likely that a great many people using Coinhive — legitimately on their own sites or otherwise — are going to lose some money as a result. That’s because Coinhive’s minimum payout is .05 Monero, which equals roughly USD $2.35.

“That means Coinhive is going to keep all the virtually currency from user accounts that have mined something below that threshold,” he said. “Maybe that’s just a few dollars or a few pennies here or there, but that’s kind of been their business model all along. They have made a lot of money through their platform.”

KrebsOnSecurity’s March 2018 Coinhive story traced the origins of the mining service back to Dominic Szablewski, a programmer who founded the German-language image board pr0gramm[.]com (not safe for work). The story noted that Coinhive began as a money-making experiment that was first debuted on the pr0gramm Web site.

The Coinhive story prompted an unusual fundraising campaign from the pr0gramm[.]com user community, which expressed alarm over the publication of details related to the service’s founders (even though all of the details included in that piece were drawn from publicly-searchable records). In an expression of solidarity to protest that publication, the pr0gramm board members collectively donated hundreds of thousands of euros to various charities that support curing cancer (Krebs is translated in German to “cancer” or “crab.”)

After that piece ran, Coinhive added to its Web site the contact information for Badges2Go UG, a limited liability company established in 2017 and headed by a Slyvia Klein from Frankfurt who is also head of an entity called Blockchain Future. Klein did not respond to requests for comment.

A Russian court has handed down lengthy prison terms for two men convicted on treason charges for allegedly sharing information about Russian cybercriminals with U.S. law enforcement officials. The men — a former Russian cyber intelligence official and an executive at Russian security firm Kaspersky Lab — were reportedly prosecuted for their part in an investigation into Pavel Vrublevsky, a convicted cybercriminal who ran one of the world’s biggest spam networks and was a major focus of my 2014 book, Spam Nation.

Sergei Mikhailov, formerly deputy chief of Russia’s top anti-cybercrime unit, was sentenced today to 22 years in prison. The court also levied a 14-year sentence against Ruslan Stoyanov, a senior employee at Kaspersky Lab. Both men maintained their innocence throughout the trial.

Following their dramatic arrests in 2016, many news media outlets reported that the men were suspected of having tipped off American intelligence officials about those responsible for Russian hacking activities tied to the 2016 U.S. presidential election.

That’s because two others arrested for treason at the same time — Mikhailov subordinates Georgi Fomchenkov and Dmitry Dokuchaev — were reported by Russian media to have helped the FBI investigate Russian servers linked to the 2016 hacking of the Democratic National Committee. The case against Fomchenkov and Dokuchaev has not yet gone to trial.

What exactly was revealed during the trial of Mikhailov and Stoyanov is not clear, as the details surrounding it were classified. But according to information first reported by KrebsOnSecurity in January 2017, the most likely explanation for their prosecution stemmed from a long-running grudge held by Pavel Vrublevsky, a Russian businessman who ran a payment firm called ChronoPay and for years paid most of the world’s top spammers and virus writers to pump malware and hundreds of billions of junk emails into U.S. inboxes.

In 2013, Vrublevsky was convicted of hiring his most-trusted spammer and malware writer to launch a crippling distributed denial-of-service (DDoS) attack against one of his company’s chief competitors.

Prior to Vrublevsky’s conviction, massive amounts of files and emails were taken from Vrublevsky’s company and shared with this author. Those included spreadsheets chock full of bank account details tied to some of the world’s most active cybercriminals, and to a vast network of shell corporations created by Vrublevsky and his co-workers to help launder the proceeds from their various online pharmacy, spam and fake antivirus operations.

In a telephone interview with this author in 2011, Vrublevsky said he was convinced that Mikhailov was taking information gathered by Russian government cybercrime investigators and feeding it to U.S. law enforcement and intelligence agencies. Vrublevsky told me then that if ever he could prove for certain Mikhailov was involved in leaking incriminating data on ChronoPay, he would have someone “tear him a new asshole.”

An email that Vrublevsky wrote to a ChronoPay employee in 2010 eerily presages the arrests of Mikhailov and Stoyanov, voicing Vrublevsky’s suspicion that the two were closely involved in leaking ChronoPay emails and documents that were seized by Mikhailov’s own division. A copy of that email is shown in Russian in the screen shot below. A translated version of the message text is available here (PDF).

Predictably, Vrublevsky has taken to gloating on Facebook about today’s prison’s sentences, calling them “good news.” He told the Associated Press that Mikhailov had abused his position at the FSB to go after Internet entrepreneurs like him and “turn them into cybercriminals,” thus “whipping up cyber hysteria around the world.”

This is a rather rich quote, as Vrublevsky was already a well-known and established cybercriminal long before Mikhailov came into his life. Also, I would not put it past Vrublevsky to have somehow greased the wheels of this prosecution.

As I noted in Spam Nation, emails leaked from ChronoPay suggest that Vrublevsky funneled as much as $1 million to corrupt Russian political leaders for the purpose of initiating a criminal investigation into Igor Gusev, a former co-founder of ChronoPay who went on to create a pharmacy spam operation that closely rivaled Vrublevsky’s own pharmacy spam operation — Rx Promotion.

Payroll software provider Apex Human Capital Management suffered a ransomware attack this week that severed payroll management services for hundreds of the company’s customers for nearly three days. Faced with the threat of an extended outage, Apex chose to pay the ransom demand and begin the process of restoring service to customers.

Roswell, Ga. based Apex HCM is a cloud-based payroll software company that serves some 350 payroll service bureaus that in turn provide payroll services to small and mid-sized businesses. At 4 a.m. on Tuesday, Feb. 19, Apex was alerted that its systems had been infected with a destructive strain of ransomware that encrypts computer files and demands payment for a digital key needed to unscramble the data.

The company quickly took all of its systems offline, and began notifying customers that it was trying to remediate a security threat. Over a series of bi-hourly updates, Apex kept estimating that it expected to restore service in a few hours, only to have to walk back those estimates almost every other time a new customer update went out.

Contacted Wednesday by an Apex client who was nervous about being unable to make this week’s payroll for his clients, KrebsOnSecurity reached out to Apex for comment. Ian Oxman, the company’s chief marketing officer, said the ransomware never touched customer data, but instead encrypted and disrupted everything in the company’s computer systems and at its off-site disaster recovery systems.

“We had just recently completed a pretty state-of-the-art disaster recovery plan off-site out and of state that was mirroring our live system,” Oxman said. “But when the ransomware bomb went off, not only did it go through and infect our own network, it was then immediately picked up in our disaster recovery site, which made switching over to that site unusable.”

Oxman said Apex hired two outside security firms, and by Feb. 20 the consensus among all three was that paying the ransom was the fastest way to get back online. The company declined to specify how much was paid or what strain of ransomware was responsible for the attack.

“We paid the ransom, and it sucked,” Oxman said. “In respect for our clients who needed to get their businesses up and running that was going to be obviously the quicker path.”

Unfortunately for Apex, paying up didn’t completely solve its problems. For one thing, Oxman said, the decryption key they were given after paying the ransom didn’t work exactly as promised. Instead of restoring all files and folders to their pre-encrypted state, the decryption process broke countless file directories and rendered many executable files inoperable — causing even more delays.

“When they encrypt the data, that happens really fast,” he said. “When they gave us the keys to decrypt it, things didn’t go quite as cleanly.”

One of Apex’s older business units — ACA OnDemand — is still offline, but the company is now offering to move customers on that platform over to newer (and more expensive) software-as-a-service systems, and to train those customers on how to use them.

Experts say attacks like the one against Apex HCM are playing out across the world every day, and have turned into a billion-dollar business for cyber thieves. The biggest group of victims are professional services firms, according to a study by NTT Security.

Ransomware victims perhaps in the toughest spot include those offering cloud data hosting and software-as-service, as these businesses are completely unable to serve their customers while a ransomware infestation is active.

The FBI and multiple security firms have advised victims not to pay any ransom demands, as doing so just encourages the attackers and in any case may not result in actually regaining access to encrypted files.

In practice, however, many cybersecurity consulting firms are quietly urging their customers that paying up is the fastest route back to business-as-usual. It’s not hard to see why: Having customer data ransomed or stolen can spell the end of cloud-based business, but just being down for more than a few days can often be just as devastating. As a result, the temptation to simply pay up may become stronger with each passing day — even if the only thing being ransomed is a bunch of desktops and servers.

On Christmas Eve 2018, cloud data hosting firm Dataresolution.net was hit with the Ryuk strain of ransomware. More than a week later on Jan. 2, 2019, this blog reported that the company — which had chosen not to pay the ransom and instead restore everything from backups — was still struggling to bring its systems back online.

One dataresolution.net client said the company didn’t succeed in rebuilding its server or turning over his company’s database stored there until Jan. 9 — 16 days after the ransomware outbreak.

“From my understanding it was another two weeks until all of the clients were rebuilt,” said the customer, who works as an IT manager at a benefits management firm that used dataresolution.net and its now transitioning away from the company. “The vendor never provided any analysis on how it occurred and how they would prevent it from occurring again.  Other than different antivirus and not allowing RDP connections to the internet they don’t seem to have put any additional safeguards in place. They did not proactively offer any compensation for the outage. I am in the process of documenting the business financial impact to request a ‘credit’ at the same time as planning on bringing the system in house.”

For its part, Apex is still trying to determine how the ransomware got into its systems.

“That’s where this forensic analysis is still going on,” Oxman said. “For us, the emergency response team literally worked 48 hours straight getting our systems back up, and secondary to that is now trying to figure out what the hell happened and how do we prevent this from happening again. We had just completed a security audit and we were feeling pretty good. Obviously, these cyber hackers found a way in, but I’m sure that’s how every company feels that gets hit.”

Here are a few tips for preventing and dealing with ransomware attacks:

-Patch, early and often: Many ransomware attacks leverage known security flaws in servers and desktops.

-Disable RDP: Short for Remote Desktop Protocol, this feature of Windows allows a system to be remotely administered over the Internet. A ridiculous number of businesses — particularly healthcare providers — get hit with ransomware because they leave RDP open to the Internet and secured with easy-to-guess passwords. And there are a number of criminal services that sell access to brute-forced RDP installations.

-Filter all email: Invest in security systems that can block executable files at the email gateway.

-Isolate mission-critical systems and data: This can be harder than it sounds. It may be worth hiring a competent security firm to make sure this is done right.

-Backup key files and databases: Bear in mind that ransomware can encrypt any network or cloud-based files or folders that are mapped and have been assigned a drive letter. Backing up to a secondary system that is not assigned a drive letter or is disconnected when it’s not backing up data is key. The old “3-2-1” backup rule comes into play here: Wherever possible, keep three backups of your data, on two different storage types, with at least one backup offsite.

-Disable macros in Microsoft Office: Block external content in Office files. Educate users that ransomware very often succeeds only when a user opens Office file attachment sent via email and manually enables Macros.

-Enable controlled folder access: Create rules to disallow the running of executable files in Windows from local user profile folders (App Data, Local App Data, ProgramData, Temp, etc.)

Sites like nomoreransom.org distribute free tools and tutorials that can help some ransomware victims recover files without paying a ransom demand, but those tools often only work with specific versions of a particular ransomware strain.

Fraud investigators say they’ve uncovered a sophisticated new breed of credit card skimmers being installed at gas pumps that is capable of relaying stolen card data via mobile text message, thereby enabling fraudsters to collect it from anywhere in the world. One interesting component of this criminal innovation is a small cellphone and Bluetooth-enabled device hidden inside the contactless payment terminal of the pump, which appears to act as a Bluetooth hub that wirelessly gathers card data from multiple compromised pumps at a given filling station.

A memo sent by the U.S. Secret Service last week to its various field offices said the agency recently was alerted to the discovery of a fraud device made to fit underneath the plastic cap for the contactless payment terminal attached to the exterior of a fuel pump. Here’s a look at the back side of that unwelcome parasite:

As we can see from the above image, it includes GSM mobile phone components, allowing it to send stolen card data wirelessly via text message. In contrast, most modern pump skimmers transmit stolen card data to the thieves via Bluetooth. The white rectangular module on the right is the mobile phone component; the much smaller, square module below and to the left is built to handle Bluetooth communications.

Bluetooth requires the fraudsters who placed the devices to return to the scene of the crime periodically and download the stolen data with a mobile device or laptop. Using SMS-based skimmers, the fraudsters never need to take that risk and can receive the stolen card data in real-time from anywhere there is mobile phone service.

Gas stations are beginning to implement contactless payments at the pump to go along with traditional magnetic stripe and chip card-based payments. These contactless payments use a technology called “near field communication,” or NFC, which exchanges wireless signals when an NFC-enabled card or mobile device is held closely to a point-of-sale device.

Because this tiny round device was found hidden inside of an NFC card reader on the outside of a gas pump, investigators said they initially thought it might have been designed to somehow siphon or interfere with data being transmitted by contactless payment cards. But this theory was quickly discarded, as contactless cards include security features which render data that might be intercepted largely useless for future transactions (or at least hardly worth the up-front investment, craftsmanship and risk it takes to deploy such skimming devices).

Mark Carl is chief executive officer at ControlScan, a company in Alpharetta, Ga. that helps merchants secure their payment card technology. Carl’s company is the one that found the skimmer and alerted local authorities, which in turn alerted the Secret Service.

Carl said his team is still trying to reverse engineer the device found inside the NFC reader at the pump, but that so far it appears its purpose is to act as a Bluetooth communications hub for other skimming devices found at the scene. Turns out, investigators also discovered traditional Bluetooth-based skimming devices attached to the power and networking cables inside various pumps at the compromised filling station.

“Based on the chipsets, and that there were other traditional skimmers in other pumps at the site, we believe this device [the round gizmo found inside the NFC reader] is likely the hub for a Bluetooth local area network,” Carl told KrebsOnSecurity. “So an attacker can install multiple skimmers in different pumps, feed all of that data to this device with Bluetooth, and then relay it all via the cellular connection.”

Many readers have asked if they should be scanning fuel pumps with their smart phones using the built-in Bluetooth component or Android mobile app like Skimmer Scanner. If this seems like fun, then by all means go right ahead, but I wouldn’t count on these methods failing to detect a Bluetooth skimmer at the pump as proof that the pump is skimmer-free.

For one thing, the skimmer detection app detects only one type of Bluetooth module used in these schemes (HC-05), and there are least three other types commonly found embedded in compromised pumps (HC-06, HC-08 and FCD_1608). And trying to do this with your mobile phone alone is not likely to yield any more conclusive results.

Better advice is to patronize filling stations that have upgraded their pumps in the past few years to add more digital and physical security features. As I wrote in last summer’s “How to Avoid Card Skimmers at the Pump,” newer and more secure pumps typically feature a horizontal card acceptance slot along with a raised metallic keypad — much like a traditional payphone keypad.

One other tip from that story: Some pump skimming devices are capable of stealing debit card PINs as well, so it’s a good idea to avoid paying with a debit card at the pump. Armed with your PIN and debit card data, thieves can clone the card and pull money out of your account at an ATM. Having your checking account emptied of cash while your bank sorts out the situation can be a huge hassle and create secondary problems (bounced checks, for instance).

This advice often runs counter to the messaging pushed by fuel station owners themselves, many of whom offer lower prices for cash or debit card transactions. That’s because credit card transactions typically are more expensive to process.

The U.S. government — along with a number of leading security companies — recently warned about a series of highly complex and widespread attacks that allowed suspected Iranian hackers to siphon huge volumes of email passwords and other sensitive data from multiple governments and private companies. But to date, the specifics of exactly how that attack went down and who was hit have remained shrouded in secrecy.

This post seeks to document the extent of those attacks, and traces the origins of this overwhelmingly successful cyber espionage campaign back to a cascading series of breaches at key Internet infrastructure providers.

Before we delve into the extensive research that culminated in this post, it’s helpful to review the facts disclosed publicly so far. On Nov. 27, 2018, Cisco’s Talos research division published a write-up outlining the contours of a sophisticated cyber espionage campaign it dubbed “DNSpionage.”

The DNS part of that moniker refers to the global “Domain Name System,” which serves as a kind of phone book for the Internet by translating human-friendly Web site names (example.com) into numeric Internet address that are easier for computers to manage.

Talos said the perpetrators of DNSpionage were able to steal email and other login credentials from a number of government and private sector entities in Lebanon and the United Arab Emirates by hijacking the DNS servers for these targets, so that all email and virtual private networking (VPN) traffic was redirected to an Internet address controlled by the attackers.

Talos reported that these DNS hijacks also paved the way for the attackers to obtain SSL encryption certificates for the targeted domains (e.g. webmail.finance.gov.lb), which allowed them to decrypt the intercepted email and VPN credentials and view them in plain text.

On January 9, 2019, security vendor FireEye released its report, “Global DNS Hijacking Campaign: DNS Record Manipulation at Scale,” which went into far greater technical detail about the “how” of the espionage campaign, but contained few additional details about its victims.

Twelve days after the FireEye report, the U.S. Department of Homeland Security issued a rare emergency directive ordering all U.S. federal civilian agencies to secure the login credentials for their Internet domain records. As part of that mandate, DHS published a short list of domain names and Internet addresses that were used in the DNSpionage campaign, although those details did not go beyond what was previously released by either Cisco Talos or FireEye.

That changed on Jan. 25, 2019, when security firm CrowdStrike published a blog post listing virtually every Internet address known to be (ab)used by the espionage campaign to date. The remainder of this story is based on open-source research and interviews conducted by KrebsOnSecurity in an effort to shed more light on the true extent of this extraordinary — and ongoing — attack.

PASSIVE DNS

I began my research by taking each of the Internet addresses laid out in the CrowdStrike report and running them through both Farsight Security and SecurityTrails, services that passively collect data about changes to DNS records tied to tens of millions of Web site domains around the world.

Working backwards from each Internet address, I was able to see that in the last few months of 2018 the hackers behind DNSpionage succeeded in compromising key components of DNS infrastructure for more than 50 Middle Eastern companies and government agencies, including targets in Albania, Cyprus, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Saudi Arabia and the United Arab Emirates.

For example, the passive DNS data shows the attackers were able to hijack the DNS records for mail.gov.ae, which handles email for government offices of the United Arab Emirates. Here are just a few other interesting assets successfully compromised in this cyber espionage campaign:

-nsa.gov.iq: the National Security Advisory of Iraq
-webmail.mofa.gov.ae: email for the United Arab Emirates’ Ministry of Foreign Affairs
-shish.gov.al: the State Intelligence Service of Albania
-mail.mfa.gov.eg: mail server for Egypt’s Ministry of Foreign Affairs
-mod.gov.eg: Egyptian Ministry of Defense
-embassy.ly: Embassy of Libya
-owa.e-albania.al: the Outlook Web Access portal for the e-government portal of Albania
-mail.dgca.gov.kw: email server for Kuwait’s Civil Aviation Bureau
-gid.gov.jo: Jordan’s General Intelligence Directorate
-adpvpn.adpolice.gov.ae: VPN service for the Abu Dhabi Police
-mail.asp.gov.al: email for Albanian State Police
-owa.gov.cy: Microsoft Outlook Web Access for Government of Cyprus
-webmail.finance.gov.lb: email for Lebanon Ministry of Finance
-mail.petroleum.gov.eg: Egyptian Ministry of Petroleum
-mail.cyta.com.cy: Cyta telecommunications and Internet provider, Cyprus
-mail.mea.com.lb: email access for Middle East Airlines

The passive DNS data provided by Farsight and SecurityTrails also offered clues about when each of these domains was hijacked. In most cases, the attackers appear to have changed the DNS records for these domains (we’ll get to the “how” in a moment) so that the domains pointed to servers in Europe that they controlled.

Shortly after the DNS records for these TLDs were hijacked — sometimes weeks, sometimes just days or hours — the attackers were able to obtain SSL certificates for those domains from SSL providers Comodo and/or Let’s Encrypt. The preparation for several of these attacks can be seen at cert.sh, which provides a searchable database of all new SSL certificate creations.

Let’s take a closer look at one example. The CrowdStrike report references the Internet address 139.59.134[.]216 (see above), which according to Farsight was home to just seven different domains over the years. Two of those domains only appeared at that Internet address in December 2018, including domains in Lebanon and — curiously — Sweden.

The first domain was “ns0.idm.net.lb,” which is a server for the Lebanese Internet service provider IDM. From early 2014 until December 2018, ns0.idm.net.lb pointed to 194.126.10[.]18, which appropriately enough is an Internet address based in Lebanon. But as we can see in the screenshot from Farsight’s data below, on Dec. 18, 2018, the DNS records for this ISP were changed to point Internet traffic destined for IDM to a hosting provider in Germany (the 139.59.134[.]216 address).

Notice what else is listed along with IDM’s domain at 139.59.134[.]216, according to Farsight:

The DNS records for the domains sa1.dnsnode.net and fork.sth.dnsnode.net also were changed from their rightful home in Sweden to the German hosting provider controlled by the attackers in December. These domains are owned by Netnod Internet Exchange, a major global DNS provider based in Sweden. Netnod also operates one of the 13 “root” name servers, a critical resource that forms the very foundation of the global DNS system.

We’ll come back to Netnod in a moment. But first let’s look at another Internet address referenced in the CrowdStrike report as part of the infrastructure abused by the DNSpionage hackers: 82.196.11[.]127. This address in The Netherlands also is home to the domain mmfasi[.]com, which Crowdstrike says was one of the attacker’s domains that was used as a DNS server for some of the hijacked infrastructure.

As we can see in the screenshot above, 82.196.11[.]127 was temporarily home to another pair of Netnod DNS servers, as well as the server “ns.anycast.woodynet.net.” That domain is derived from the nickname of Bill Woodcock, who serves as executive director of Packet Clearing House (PCH).

PCH is a nonprofit entity based in northern California that also manages significant amounts of the world’s DNS infrastructure, particularly the DNS for more than 500 top-level domains and a number of the Middle East top-level domains targeted by DNSpionage.

TARGETING THE REGISTRARS

Contacted on Feb. 14 by KrebsOnSecurity, Netnod CEO Lars Michael Jogbäck confirmed that parts of Netnod’s DNS infrastructure were hijacked in late December 2018 and early January 2019 after the attackers gained access to accounts at Netnod’s domain name registrar.

Jogbäck pointed to a statement the company published on its Web site on Feb. 5, which says Netnod learned of its role in the attack on January 2 and has been in contact with all relevant parties and customers throughout this process.

“As a participant in an international security co-operation, Netnod became aware on 2 January 2019 that we had been caught up in this wave and that we had experienced a MITM (man-in-the-middle) attack,” the statement reads. “Netnod was not the ultimate goal of the attack. The goal is considered to have been the capture of login details for Internet services in countries outside of Sweden.”

In an interview with this author on Feb. 15, PCH’s Woodcock acknowledged that portions of his organization’s DNS infrastructure were compromised after the DNSpionage hackers abused unauthorized access to its domain name registrar.

As it happens, the registrar records for both pch.net and dnsnode.net point to the same sources: Key-Systems GmbH, a domain registrar based in Germany; and Frobbit.se, a company in Sweden. Frobbit is a reseller of Key Systems, and the two companies share some of the same online resources.

Woodcock said domain records for the targeted Middle East TLDs it managed were altered after the DNSpionage hackers phished credentials that Key-Systems uses to make domain changes for their clients.

Specifically, he said, the hackers phished credentials that PCH’s registrar used to send signaling messages known as the Extensible Provisioning Protocol (EPP). EPP is a little-known interface that serves as a kind of back-end for the global DNS system, allowing domain registrars to notify the regional registries (like Verisign) about changes to domain records, including new domain registrations, modifications, and transfers.

“At the beginning of January, Key-Systems said they believed that their EPP interface had been abused by someone who had stolen valid credentials,” Woodcock said.

Key-Systems declined to comment for this story, beyond saying it does not discuss details of its reseller clients’ businesses.

Netnod’s written statement on the attack referred further inquiries to the company’s security director Patrik Fältström, who also is co-owner of Frobbit.se.

In an email to KrebsOnSecurity, Fältström said unauthorized EPP instructions were sent to various registries by the DNSpionage attackers from both Frobbit and Key Systems.

“The attack was from my perspective clearly an early version of a serious EPP attack,” he wrote. “That is, the goal was to get the right EPP commands sent to the registries. I am extremely nervous personally over extrapolations towards the future. Should registries allow any EPP command to come from the registrars? We will always have some weak registrars, right?”

DNSSEC

One of the more interesting aspects of these attacks is that both Netnod and PCH are vocal proponents and adopters of DNSSEC (a.k.a. “DNS Security Extensions”), which is a technology designed to defeat the very type of attack that the DNSpionage hackers were able to execute.

DNSSEC protects applications from using forged or manipulated DNS data, by requiring that all DNS queries for a given domain or set of domains be digitally signed. In DNSSEC, if a name server determines that the address record for a given domain has not been modified in transit, it resolves the domain and lets the user visit the site. If, however, that record has been modified in some way or doesn’t match the domain requested, the name server blocks the user from reaching the fraudulent address.

While DNSSEC can be an effective tool for mitigating attacks such as those launched by DNSpionage, only about 20 percent of the world’s major networks and Web sites have enabled it, according to measurements gathered by APNIC, the regional Internet address registry for the Asia-Pacific region.

Jogbäck said Netnod’s infrastructure suffered three separate attacks from the DNSpionage attackers. The first two occurred in a two-week window between Dec. 14, 2018 and Jan. 2, 2019, and targeted company servers that were not protected by DNSSEC.

However, he said the third attack between Dec. 29 and Jan. 2 targeted Netnod infrastructure that was protected by DNSSEC and serving its own internal email network. Yet, because the attackers already had access to its registrar’s systems, they were able to briefly disable that safeguard — or at least long enough to obtain SSL certificates for two of Netnod’s email servers.

Jogbäck told KrebsOnSecurity that once the attackers had those certificates, they re-enabled DNSSEC for the company’s targeted servers while apparently preparing to launch the second stage of the attack — diverting traffic flowing through its mail servers to machines the attackers controlled. But Jogbäck said that for whatever reason, the attackers neglected to use their unauthorized access to its registrar to disable DNSSEC before later attempting to siphon Internet traffic.

“Luckily for us, they forgot to remove that when they launched their man-in-the-middle attack,” he said. “If they had been more skilled they would have removed DNSSEC on the domain, which they could have done.”

Woodcock says PCH validates DNSSEC on all of its infrastructure, but that not all of the company’s customers — particularly some of the countries in the Middle East targeted by DNSpionage — had configured their systems to fully implement the technology.

Woodcock said PCH’s infrastructure was targeted by DNSpionage attackers in four distinct attacks between December 13, 2018 and January 2, 2019. With each attack, the hackers would turn on their password-slurping tools for roughly one hour, and then switch them off before returning the network to its original state after each run.

The attackers didn’t need to enable their surveillance dragnet longer than an hour each time because most modern smartphones are configured to continuously pull new email for any accounts the user may have set up on his device. Thus, the attackers were able to hoover up a great many email credentials with each brief hijack.

On Jan. 2, 2019 — the same day the DNSpionage hackers went after Netnod’s internal email system — they also targeted PCH directly, obtaining SSL certificates from Comodo for two PCH domains that handle internal email for the company.

Woodcock said PCH’s reliance on DNSSEC almost completely blocked that attack, but that it managed to snare email credentials for two employees who were traveling at the time. Those employees’ mobile devices were downloading company email via hotel wireless networks that — as a prerequisite for using the wireless service — forced their devices to use the hotel’s DNS servers, not PCH’s DNNSEC-enabled systems.

“The two people who did get popped, both were traveling and were on their iPhones, and they had to traverse through captive portals during the hijack period,” Woodcock said. “They had to switch off our name servers to use the captive portal, and during that time the mail clients on their phones checked for new email. Aside from that, DNSSEC saved us from being really, thoroughly owned.”

Because PCH had protected its domains with DNSSEC, the practical effect of the hijack against its mail infrastructure was that for roughly an hour nobody but the two remote employees received any email.

“For essentially all of our users, what it looked like was the mail server just wasn’t available for a short period,” Woodcock said. “It didn’t resolve for a while if they happened to be checking their phone or whatever, and each person thought well that’s funny, I’ll check it back in a while. And by the time they checked again it was working fine. A bunch of our staff noticed a brief outage in our email service, but nobody thought enough of it to discuss it with anyone else or open a ticket.”

But the DNSpionage hackers were not deterred. In a letter to its customers sent earlier this month, PCH said a forensic investigation determined that on Jan. 24 a computer which holds its Web site user database had been compromised. The user data stored in the database included customer usernames, bcrypt password hashes, emails, addresses, and organization names.

“We see no evidence that the attackers accessed the user database or exfiltrated it,” the message reads. “So we are providing you this information as a matter of transparency and precaution, rather than because we believe that your data was compromised.”

IMPROVEMENTS

Multiple experts interviewed for this story said one persistent problem with DNS-based attacks is that a great deal of organizations tend to take much of their DNS infrastructure for granted. For example, many entities don’t even log their DNS traffic, nor do they keep a close eye on any changes made to their domain records.

Even for those companies making an effort to monitor their DNS infrastructure for suspicious changes, some monitoring services only take snapshots of DNS records passively, or else only do so actively on a once-daily basis. Indeed, Woodcock said PCH relied on no fewer than three monitoring systems, and that none of them alerted his organization to the various one-hour hijacks that hit PCH’s DNS systems.

“We had three different commercial DNS monitoring services, none of which caught it,” he said. “None of them even warned us that it had happened after the fact.”

Woodcock said PCH has since set up a system to poll its own DNS infrastructure multiple times each hour, and to alert immediately on any changes.

Jogbäck said Netnod also has beefed up its monitoring, as well as redoubled efforts to ensure that all of the available options for securing their domain infrastructure were being used. For instance, the company had not previously secured all of its domains with a “domain lock,” a service that requires a registrar to take additional authentication steps before making any modifications to a domain’s records.

“We are really sad we didn’t do a better job of protecting our customers, but we are also a victim in the chain of the attack,” Jogbäck said. “You can change to a better lock after you’ve been robbed, and hopefully make it more difficult for someone to do it again. But I can truly say we have learned a tremendous amount from being a victim in this attack, and we are now much better off than before.”

Woodcock said he’s worried that Internet policymakers and other infrastructure providers aren’t taking threats to the global DNS seriously or urgently enough, and he’s confident the DNSpionage hackers will have plenty of other victims to target and exploit in the months and years ahead.

“All of this is a running battle,” he said. “The Iranians are not just trying to do these attacks to have an immediate effect. They’re trying to get into the Internet infrastructure deeply enough so they can get away with this stuff whenever they want to. They’re looking to get as many ways in as possible that they can use for specific goals in the future.”

RECOMMENDATIONS

John Crain is chief security, stability and resiliency officer at ICANN, the non-profit entity that oversees the global domain name industry. Crain said many of the best practices that can make it more difficult for attackers to hijack a target’s domains or DNS infrastructure have been known for more than a decade.

“A lot of this comes down to data hygiene,” Crain said. “Large organizations down to mom-and-pop entities are not paying attention to some very basic security practices, like multi-factor authentication. These days, if you have a sub-optimal security stance, you’re going to get owned. That’s the reality today. We’re seeing much more sophisticated adversaries now taking actions on the Internet, and if you’re not doing the basic stuff they’re going to hit you.”

Some of those best practices for organizations include:

-Use DNSSEC (both signing zones and validating responses)

-Use registration features like Registry Lock that can help protect domain names records from being changed

-Use access control lists for applications, Internet traffic and monitoring

-Use 2-factor authentication, and require it to be used by all relevant users and subcontractors

-In cases where passwords are used, pick unique passwords and consider password managers

-Review accounts with registrars and other providers

-Monitor certificates by monitoring, for example, Certificate Transparency Logs