The ongoing breach affecting thousands of organizations that relied on backdoored products by network software firm SolarWinds may have jeopardized the privacy of countless sealed court documents on file with the U.S. federal court system, according to a memo released Wednesday by the Administrative Office (AO) of the U.S. Courts.

The judicial branch agency said it will be deploying more stringent controls for receiving and storing sensitive documents filed with the federal courts, following a discovery that its own systems were compromised as part of the SolarWinds supply chain attack. That intrusion involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for some 18,000 users of its Orion network management software as far back as March 2020.

“The AO is working with the Department of Homeland Security on a security audit relating to vulnerabilities in the Judiciary’s Case Management/Electronic Case Files system (CM/ECF) that greatly risk compromising highly sensitive non-public documents stored on CM/ECF, particularly sealed filings,” the agency said in a statement published Jan. 6.

“An apparent compromise of the confidentiality of the CM/ECF system due to these discovered vulnerabilities currently is under investigation,” the statement continues. “Due to the nature of the attacks, the review of this matter and its impact is ongoing.”

The AO declined to comment on specific questions about their breach disclosure. But a source close to the investigation told KrebsOnSecurity that the federal court document system was “hit hard,” by the SolarWinds attackers, which multiple U.S. intelligence and law enforcement agencies have attributed as “likely Russian in origin.”

The source said the intruders behind the SolarWinds compromise seeded the AO’s network with a second stage “Teardrop” malware that went beyond the “Sunburst” malicious software update that was opportunistically pushed out to all 18,000 customers using the compromised Orion software. This suggests the attackers were targeting the agency for deeper access to its networks and communications.

The AO’s court document system powers a publicly searchable database called PACER, and the vast majority of the files in PACER are not restricted and are available to anyone willing to pay for the records.

But experts say many other documents stored in the AO’s system are sealed — either temporarily or indefinitely by the courts or parties to a legal matter — and may contain highly sensitive information, including intellectual property and trade secrets, or even the identities of confidential informants.

Nicholas Weaver, a lecturer at the computer science department at University of California, Berkeley, said the court document system doesn’t hold documents that are classified for national security reasons. But he said the system is full of sensitive sealed filings — such as subpoenas for email records and so-called “trap and trace” requests that law enforcement officials use to determine with whom a suspect is communicating via phone, when and for how long.

“This would be a treasure trove for the Russians knowing about a lot of ongoing criminal investigations,” Weaver said. “If the FBI has indicted someone but hasn’t arrested them yet, that’s all under seal. A lot of the investigative tools that get protected under seal are filed very early on in the process, often with gag orders that prevent [the subpoenaed party] from disclosing the request.”

The acknowledgement from the AO comes hours after the U.S. Justice Department said it also was a victim of the SolarWinds intruders, who took control over the department’s Office 365 system and accessed email sent or received from about three percent of DOJ accounts (the department has more than 100,000 employees).

The SolarWinds hack also reportedly jeopardized email systems used by top Treasury Department officials, and granted the attackers access to networks inside the Energy, Commerce and Homeland Security departments.

The New York Times on Wednesday reported that investigators are examining whether a breach at another software provider — JetBrains — may have precipitated the attack on SolarWinds. The company, which was founded by three Russian engineers in the Czech Republic, makes a tool called TeamCity that helps developers test and manage software code. TeamCity is used by developers at 300,000 organizations, including SolarWinds and 79 of the Fortune 100 companies.

“Officials are investigating whether the company, founded by three Russian engineers in the Czech Republic with research labs in Russia, was breached and used as a pathway for hackers to insert back doors into the software of an untold number of technology companies,” The Times said. “Security experts warn that the monthslong intrusion could be the biggest breach of United States networks in history.”

Under the AO’s new procedures, highly sensitive court documents filed with federal courts will be accepted for filing in paper form or via a secure electronic device, such as a thumb drive, and stored in a secure stand-alone computer system. These sealed documents will not be uploaded to CM/ECF.

“This new practice will not change current policies regarding public access to court records, since sealed records are confidential and currently are not available to the public,” the AO said.

James Lewis, senior vice president at the Center for Strategic and International Studies, said it’s too soon to tell the true impact of the breach at the court system, but the fact that they were apparently targeted is a “a very big deal.”

“We don’t know what the Russians took, but the fact that they had access to this system means they had access to a lot of great stuff, because federal cases tend to involve fairly high profile targets,” he said.

Like countless others, I frittered away the better part of Jan. 6 doomscrolling and watching television coverage of the horrifying events unfolding in our nation’s capital, where a mob of President Trump supporters and QAnon conspiracy theorists was incited to lay siege to the U.S. Capitol. For those trying to draw meaning from the experience, might I suggest consulting the literary classic Moby Dick, which simultaneously holds clues about QAnon’s origins and offers an apt allegory about a modern-day Captain Ahab and his ill-fated obsessions.

Many have speculated that Jim Watkins, the administrator of the online message board 8chan (a.k.a. 8kun), and/or his son Ron are in fact “Q,” the anonymous persona behind the QAnon conspiracy theory, which holds that President Trump is secretly working to save the world from a satanic cult of pedophiles and cannibals.

Last year, as I was scrutinizing the computer networks that kept QAnon online, researcher Ron Guilmette pointed out a tantalizing utterance from Watkins the younger which adds tenuous credence to the notion that one or both of them is Q.

We’ll get to how the Great White Whale (the Capitol?) fits into this tale in a moment. But first, a bit of background. A person identified only as “Q” has for years built an impressive following for the far-right conspiracy movement by leaving periodic “Q drops,” cryptic messages that QAnon adherents spend much time and effort trying to decipher and relate to current events.

Researchers who have studied more than 5,000 Q drops are convinced that there are two distinct authors of these coded utterances. The leading theory is that those identities corresponded to the aforementioned father-and-son team responsible for operating 8chan.

Jim Watkins, 56, is the current owner of 8chan, a community perhaps now best known as a forum for violent extremists and mass shooters. Watkins is an American pig farmer based in the Philippines; Ron reportedly resides in Japan.

In the aftermath of back-to-back mass shootings on Aug. 3 and Aug. 4, 2019 in which a manifesto justifying one of the attacks was uploaded to 8chan, Cloudflare stopped providing their content delivery network to 8chan. Several other providers quickly followed suit, leaving 8chan offline for months before it found a haven at a notorious bulletproof hosting facility in Russia.

One reason Q watchers believe Ron and Jim Watkins may share authorship over the Q drops is that while 8chan was offline, the messages from Q ceased. The drops reappeared only months later when 8chan rebranded as 8kun.

CALL ME ISHMAEL

Here’s where the admittedly “Qonspiratorial” clue about the Watkins’ connection to Q comes in. On Aug. 5, 2019, Ron Watkins posted a Twitter message about 8chan’s ostracization which compared the community’s fate to that of the Pequod, the name of the doomed whaling ship in the Herman Melville classic “Moby Dick.”

“If we are still down in a few hours then maybe 8chan will just go clearnet and we can brave DDOS attacks like Ishmael on the Pequod,” Watkins the younger wrote.

Ishmael, the first-person narrator in the novel, is a somewhat disaffected American sailor who decides to try his hand at a whaling ship. Ishmael is a bit of a minor character in the book; very soon into the novel we are introduced to a much more interesting and enigmatic figure — a Polynesian harpooner by the name of Queequeg.

Apart from being a cannibal from the Pacific islands who has devoured many people, Queequeg is a pretty nice guy and shows Ismael the ropes of whaling life. Queequeg is covered head to toe in tattoos, which are described by the narrator as the work of a departed prophet and seer from the cannibal’s home island.

Like so many Q drops, Queequeg’s tattoos tell a mysterious tale, but we never quite learn what that full story is. Indeed, the artist who etched them into Queequeg’s body is long dead, and the cannibal himself can’t seem to explain what it all means.

Ishmael describes Queequeg’s mysterious markings in this passage:

“…a complete theory of the heavens and earth, and a mystical treatise on the art of attaining truth; so that Queequeg in his own proper person was a riddle to unfold; a wondrous work in one volume; but whose mysteries not even himself could read, though his own live heart beat against them; and these mysteries were therefore destined in the end to moulder away with the living parchment whereon they were inscribed, and so be unsolved to the last.”

THE GREAT WHITE WHALE

It’s perhaps fitting then that one of the most recognizable figures from the mob that stormed the U.S. Capitol on Wednesday was a heavily-tattooed, spear-wielding QAnon leader who goes by the name “Q Shaman” (a.k.a. Jake Angeli).

“Angeli’s presence at the riot, along with others wearing QAnon paraphernalia, comes as the conspiracy-theory movement has been responsible for the popularization of Trump’s voter-fraud conspiracy theories,” writes Rachel E. Greenspan for Yahoo! News.

“As Q has become increasingly hands-off, giving fewer and fewer messages to his devotees, QAnon leaders like Angeli have gained fame and power in the movement,” Greenspan wrote.

If somehow Moby Dick was indeed the inspiration for the “Q” identity in QAnon, yesterday’s events at The Capitol were the inexorable denouement of a presidential term that increasingly came to be defined by conspiracy theories. In a somewhat prescient Hartford Courant op-ed published in 2018, author Steven Almond observed that Trump’s presidency could be best understood through the lens of the Pequod’s Captain Ahab. To wit:

“Melville is offering a mythic account of how one man’s virile bombast ensnares everyone and everything it encounters. The setting is nautical, the language epic. But the tale, stripped to its ribs, is about the seductive power of the wounded male ego, how naturally a ship steered by men might tack to its vengeful course.”

“Trump’s presidency has been, in its way, a retelling of this epic. Whether we cast him as agent or principal hardly matters. What matters is that Americans have joined the quest. In rapture or disgust, we’ve turned away from the compass of self-governance and toward the mesmerizing drama of aggression on display, the masculine id unchained and all that it unchains within us. With every vitriolic tweet storm and demeaning comment, Trump strikes through the mask.”

EPILOGUE

If all of the above theorizing reads like yet another crackpot QAnon conspiracy, that may be the inevitable consequence of my spending far too much time going down this particular rabbit hole (and re-reading Moby Dick in the process!).

In any case, none of this is likely to matter to the diehard QAnon conspiracy theorists themselves, says Mike Rothschild, a writer who specializes in researching and debunking conspiracy theories.

“Even if Jim Watkins was revealed as owning the board or making the posts, it wouldn’t matter,” Rothschild said. “Anything that happens that disconfirms Q being an official in the military industrial complex is going to help fuel their persecution complex.”

Rothschild has been working hard on finishing his next book, “The Storm is Upon Us: How QAnon Became a Movement, Cult, and Conspiracy Theory of Everything,” which is due to be published in October 2021. Who’s printing the book? Ten points if you guessed Melville House, an independent publisher named after Herman Melville.

In October 2020, KrebsOnSecurity looked at how a web of sites connected to conspiracy theory movements QAnon and 8chan were being kept online by DDoS-Guard, a dodgy Russian firm that also hosts the official site for the terrorist group Hamas. New research shows DDoS-Guard relies on data centers provided by a U.S.-based publicly traded company, which experts say could be exposed to civil and criminal liabilities as a result of DDoS-Guard’s business with Hamas.

Last year’s story examined how a phone call to Oregon-based CNServers was all it took to briefly sideline multiple websites related to 8chan/8kun — a controversial online image board linked to several mass shootings — and QAnon, the far-right conspiracy theory which holds that a cabal of Satanic pedophiles is running a global child sex-trafficking ring and plotting against President Donald Trump.

From that piece:

A large number of 8kun and QAnon-related sites (see map above) are connected to the Web via a single Internet provider in Vancouver, Wash. called VanwaTech (a.k.a. “OrcaTech“). Previous appeals to VanwaTech to disconnect these sites have fallen on deaf ears, as the company’s owner Nick Lim reportedly has been working with 8kun’s administrators to keep the sites online in the name of protecting free speech.

After that story, CNServers and a U.K.-based hosting firm called SpartanHost both cut ties with VanwaTech. Following a brief disconnection, the sites came back online with the help of DDoS-Guard, an Internet company based in Russia. DDoS-Guard is now VanwaTech’s sole connection to the larger Internet.

A review of the several thousand websites hosted by DDoS-Guard is revelatory, as it includes a vast number of phishing sites and domains tied to cybercrime services or forums online.

Replying to requests for comment from a CBSNews reporter following up on my Oct. 2020 story, DDoS-Guard issued a statement saying, “We observe network neutrality and are convinced that any activity not prohibited by law in our country has the right to exist.”

But experts say DDoS-Guard’s business arrangement with a Denver-based publicly traded data center firm could create legal headaches for the latter thanks to the Russian company’s support of Hamas.

In a press release issued in late 2019, DDoS-Guard said its services rely in part on a traffic-scrubbing facility in Los Angeles owned by CoreSite [NYSE:COR], a real estate investment trust which invests in “carrier-neutral data centers and provides colocation and peering services.”

Hamas has long been named by the U.S. Treasury and State departments as a Specially Designated Global Terrorist (SDGT) organization. Under such a designation, any U.S. person or organization that provides money, goods or services to an SDGT entity could face civil and/or criminal prosecution and hefty fines ranging from $250,000 to $1 million per violation.

Sean Buckley, a former Justice Department prosecutor with the law firm Kobre & Kim, said U.S. persons and companies within the United States “are prohibited from any transaction or dealing in property or interests in property blocked pursuant to an entity’s designation as a SDGT, including but not limited to the making or receiving of any contribution of funds, goods, or services to or for the benefit of individuals or entities so designated.”

CoreSite did not respond to multiple requests for comment. But Buckley said companies can incur fines and prosecution for violating SDGT sanctions even when they don’t know that they are doing so.

In 2019, for example, a U.S. based cosmetics company was fined $1 million after investigators determined its eyelash kits were sourcing materials from North Korea, even though the supplier in that case told the cosmetics firm the materials had come from China.

“U.S. persons or companies found to willfully violate these regulations can be subject to criminal penalties under the International Emergency Economic Powers Act,” Buckley said. “However, even in the case that they are unaware they’re violating these regulations, or if the transaction isn’t directly with the sanctioned entity, these companies still run a risk of facing substantial civil and monetary penalties by the Department of Treasury’s Office of Foreign Asset Control if the sanctioned entity stands to benefit from such a transaction.”

DDoS-Guard said its partnership with CoreSite will help its stable of websites load more quickly and reliably for people visiting them from the United States. It is possible that when and if CoreSite decides it’s too risky to continue doing business with DDoS-Guard, sites like those affiliated with Hamas, QAnon and 8Chan may become more difficult to reach.

Meanwhile, DDoS-Guard customer VanwaTech continues to host a slew of sites promoting the conspiracy theory that the U.S. 2020 presidential election was stolen from President Donald Trump via widespread voting fraud and hacked voting machines, including maga[.]host, donaldsarmy[.]us, and donaldwon[.]com.

These sites are being used to help coordinate a protest rally in Washington, D.C. on January 6, 2021, the same day the U.S. Congress is slated to count electoral votes certified by the Electoral College, which in December elected Joseph R. Biden as the 46th president of The United States.

In a tweet late last year, President Trump urged his supporters to attend the Jan. 6 protest, saying the event “will be wild.”

8chan, which has rebranded as 8kun, has been linked to white supremacism, neo-Nazism, antisemitism, multiple mass shootings, and child pornography. The FBI in 2019 identified QAnon as a potential domestic terror threat, noting that some of its followers have been linked to violent incidents motivated by fringe beliefs.

Today marks the 11th anniversary of KrebsOnSecurity! Thank you, Dear Readers, for your continued encouragement and support!

With the ongoing disruption to life and livelihood wrought by the Covid-19 pandemic, 2020 has been a fairly horrid year by most accounts. And it’s perhaps fitting that this was also a leap year, piling on an extra day to a solar rotation that most of us probably can’t wait to see in the rearview mirror.

But it was hardly a dull one for computer security news junkies. In almost every category — from epic breaches and ransomware to cybercrime justice and increasingly aggressive phishing and social engineering scams — 2020 was a year that truly went to eleven.

Almost 150 stories here this past year generated nearly 9,000 responses from readers (although about 6 percent of those were on just one story). Thank you all for your thoughtful engagement, wisdom, news tips and support.

I’d like to reprise a note from last year’s anniversary post concerning ads. A good chunk of the loyal readers here are understandably security- and privacy-conscious, and many block advertisements by default — including the ads displayed here.

KrebsOnSecurity does not run third-party ads and has no plans to change that; all of the creatives you see on this site are hosted in-house, are purely image-based, and are vetted first by Yours Truly. Love them or hate ’em, these ads help keep the content at KrebsOnSecurity free to any and all readers. If you’re currently blocking ads here, please consider making an exception for this site.

In case you missed them, some of the most popular feature/enterprise stories on the site this year (in no particular order) included:

The Joys of Owning an ‘OG’ Email Account
Confessions of an ID Theft Kingpin (Part II)
Why and Where You Should Plant Your Flag
Thinking of a Career in Cybersecurity? Read This
Turn on MFA Before Crooks Do it for You
Romanian Skimmer Gang in Mexico Outed by KrebsOnSecurity Stole $1.2 Billion
Who’s Behind the ‘Web Listings’ Mail Scam?
When in Doubt: Hang Up, Look Up, & Call Back
Riding the State Unemployment Fraud Wave
Would You Have Fallen for this Phone Scam?

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2002-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cyber Security and Infrastructure Security Agency (CSIA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CSIA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

CSIA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CSIA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CSIA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”