This week, nationwide beauty products chain Sally Beauty disclosed that, for the second time in a year, it was investigating reports that hackers had broken into its networks and stolen customer credit card data. That investigation is ongoing, but I recently had an opportunity to interview a former Sally Beauty IT technician who provided a first-hand look at how the first breach in 2014 went down.

On March 14, 2014, KrebsOnSecurity broke the news that some 260,000 credit cards stolen from Sally Beauty stores had gone up for sale on Rescator[dot]cc, the same shop that first debuted cards stolen in the Home Depot and Target breaches. The company said thieves made off with just 25,000 customer cards. But the shop selling the cards listed each by the ZIP code of the Sally Beauty store from which the card data had been stolen, exactly like this same shop did with Home Depot and Target. An exhaustive analysis of the ZIP codes represented in the cards for sale on the fraud shop indicated that the hackers had hit virtually all 2,600 Sally Beauty locations nationwide.

The company never disclosed additional details about the breach itself or how it happened. But earlier this week I spoke with Blake Curlovic, until recently an application support analyst at Sally Beauty who was among the first to respond when virtual alarm bells starting going off last year about a possible intrusion. Curlovic said that at the time, Sally Beauty was running exactly one enterprise solution for security — Tripwire (full disclosure: Tripwire is an advertiser on this blog). Tripwire’s core product monitors key operating system and application files for any changes, which then triggers alerts.

Tripwire fired a warning when the intruders planted a new file on point-of-sale systems within Sally Beauty’s vast network of cash registers. The file was a program designed to steal card numbers as they were being swiped through the registers, and the attackers had named their malware after a legitimate program running on all Sally Beauty registers. They also used a utility called Timestomp to change the date and time stamp on their malware to match the legitimate file, but that apparently didn’t fool Tripwire.

According to Curlovic, the intruders gained access through a Citrix remote access portal set up for use by employees who needed access to company systems while on the road.

“The attackers somehow had login credentials of a district manager,” Curlovic said. “This guy was not exactly security savvy. When we got his laptop back in, we saw that it had his username and password taped to the front of it.”

Once inside the Sally Beauty corporate network, the attackers scanned and mapped out the entire thing, located all shared drives and scoured those for Visual Basic (VB) scripts. Network administrators in charge of managing thousands or tens of thousands of systems often will write VB scripts to automate certain tasks across all of those systems, and very often those scripts will contain usernames and passwords that can be quite useful to attackers.

Curlovic said the intruders located a VB script on Sally Beauty’s network that contained the username and password of a network administrator at the company.

“That allowed them to basically copy files to the cash registers,” he said. “They used a simple batch file loop, put in all the [cash] register Internet addresses they found while scanning the network, looped through there and copied [the malware] to all of the point-of-sale devices — roughly 6,000 of them. They were in the network for like a week prior to that planning the attack.”

HIDING IN PLAIN SIGHT

Curlovic said the malware planted on Sally Beauty’s network was identified (by some security vendors) as a variant of FrameworkPOS, a card-stealing program that exfiltrates data from the target’s network by transmitting it as domain name system (DNS) traffic.

DNS is the fundamental Internet technology that translates human-friendly domain names like example.com to numeric Internet addresses that are easier for computers to understand. All networks rely on DNS to help direct users as they surf online, but few organizations actually keep detailed logs or records of the DNS traffic traversing their networks — making it an ideal way to siphon data from a hacked network.

According to a writeup of FrameworkPOS by G Data, a security firm based in Germany, the card-stealing malware allows the attackers to dynamically configure the domain name to which the DNS traffic carrying the stolen card data will reach out. On top of that, the malware obfuscates the card data with a simple cipher so that it won’t be immediately obvious as card data to anyone who happens to be examining the DNS traffic.

But Curlovic said despite its clever data-stealing methods, other parts of the malware were clumsily written. In fact, he said, one component of the malware actually broke the Net Logon service on infected point-of-sale systems, limiting the ability of the Sally Beauty cash registers to communicate with the rest of the company’s internal network. Net Logon is a Microsoft Windows component that verifies network log on requests.

“I don’t know technically what went wrong with their software, but Net Logon wouldn’t start anymore after it was installed,” Curlovic said. “We couldn’t log in remotely with domain credentials and registers couldn’t communicate out through DNS effectively after that. It was pretty huge indicator that something was seriously wrong at that point.”

As for Sally Beauty’s standing claim that only 25,000 customer cards were taken, Curlovic said he’d be surprised if it was limited to the 260,000 originally reported by this blog.

“From what I saw in the information that the Secret Service had, 260,000 was probably on the low end,” he said. “For the period of time the software was out on the registers and running, it should have been closer to around a million, based on the number of credit transactions Sally Beauty had daily. But since the malware really wasn’t working very well, it was only capturing a portion of the cards that went through because of the formatting issue that broke the Net Logon service.

ANTI-AMERICAN MESSAGES

Curlovic said the malware used in the 2014 Sally Beauty breach communicated the stolen card data to several domains that were hosted in Ukraine, and that those domains mostly carried names that seemed to be crafted as verbal jabs at the United States.

Curlovic said he can’t recall exactly what the domains were, since he no longer has access to his notes at work, but that one of the domains was something close to “vx.anti-usa-proxy-war[dot]com.” Curlovic said he no longer has access to his notes because he was terminated from Sally Beauty about three weeks ago for reasons his former employer declined to share. He said he found out through a person at the local Denton, Texas unemployment office that someone in the company had accused him of accessing another employee’s computer without authorization. “That’s a pretty strange accusation to make, since the network administrator guys there all have access to everyone’s system on the network.”

In any case, the anti-US domains referenced by the card-stealing malware reinforce a suspicion long held by this author and other researchers: That the Sally Beauty breach was carried out by the same Russian and Ukrainian organized crime gang that stole more than 100 million credit and debit cards from both Home Depot and Target.

As I noted in a Sept. 7, 2014 story, the malware used in the Home Depot breach included several interesting text strings that chastised the United States for its role in foreign conflicts, particularly in Libya and Ukraine.

“Three of the links point to news, editorial articles and cartoons that accuse the United States of fomenting war and unrest in the name of Democracy in Ukraine, Syria, Egypt and Libya. One of the images shows four Molotov cocktails with the flags of those four nations on the bottles, next to a box of matches festooned with the American flag and match ready to strike. Another link leads to an image of the current armed conflict in Ukraine between Ukrainian forces and pro-Russian separatists.”

“This is interesting given what we know about Rescator, the individual principally responsible for running the store that is selling all of these stolen credit and debit cards. In the wake of the Target breach, I traced a long list of clues from Rescator’s various online identities back to a young programmer in Odessa, Ukraine. In his many personas, Rescator identified himself as a member of the Lampeduza cybercrime forum, and indeed this site is where he alerts customers about new batches of stolen cards.”

“As I discovered in my profile of Rescator, he and his crew seemed somewhat taken with the late despotic Libyan leader Muammar Gaddafi, although they prefer the phonetic spelling of his name. The Web site kaddafi[dot]hk was among four main carding shops run by Rescator’s crew (it has since been retired and merged with Rescator[dot]cc). The domain kaddafi[dot]me was set up to serve as an instant message Jabber server for cybercrooks, advertising its lack of logging and record keeping as a reason crooks should trust kaddafi[dot]me to handle their private online communications.”

“When I reached out to Rescator last December to obtain comment about my findings on his apparent role in the Target break-in, I received an instant message reply from the Jabber address “kaddafi@kaddafi[dot]me” (in that conversation, the person chatting with me from that address offered to pay me $10,000 if I did not run that story; I declined). But I also discovered that the kaddafi[dot]me domain was a blog of sorts that hosted some harsh and frankly chilling anti-American propaganda.”

Curlovic said the incident response team cleaning up the 2014 breach at Sally Beauty found another curious clue in the malware that attackers planted on the point-of-sale devices. They discovered that the intruders had created two versions of the card-stealing malware — one designed for use on 32-bit Windows systems and another created for use on 64-bit versions of Windows. The authors of the malware had taken the time to add an icon to the 32-bit version of the program that could be seen if anyone opened the directory where the malware was placed: The icon was little more than a black background with “Res” written in white lettering.

This kind of signing also was seen in the malware used in the Target intrusion, which contained the following text string: ““z:\Projects\Rescator\uploader\Debug\scheck.pdb”.

Normally, if one wishes to buy stolen account credentials for paid online services like Netflix, Hulu, XBox Live or Spotify, the buyer needs to visit a cybercrime forum or drop into a dark Web marketplace that only accepts Bitcoin as payment. Increasingly, however, these accounts are showing up for sale at Payivy[dot]com, an open Web marketplace that happily accepts PayPal in exchange for a variety of stolen accounts.

Marketed and sold by a Hackforums user named “Sh1eld” as a supposed method of selling ebooks and collecting payments for affiliate marketers, PayIvy has instead become a major conduit for hawking stolen accounts and credentials for a range of top Web services.

There is no central index of items for sale via PayIvy per se, but this catalog of cached sales threads offers a fairly representative glimpse: License keys for Adobe and Microsoft software products, user account credentials in bulk for services like Hulu, Netflix, Spotify, DirecTV and HBO Go, as well as a raft of gaming accounts at Origin, Steam, PlayStation and XBox Live. Other indexes at archive.is and PayIvy’s page at Reddit reveal similar results.

It’s not clear how or why PayPal isn’t shutting down most of these merchants, but some of the sellers clearly are testing things to see how far they can push it: In just five minutes of searching online, I found several PayIvy sellers who were accepting PayPal payments via PayIvy for…wait for it…hijacked PayPal accounts! The fact that PayIvy takes PayPal as payment means that buyers can purchase hacked accounts with [stolen] credit cards — or, worse yet, stolen PayPal accounts.

Jack Christin, Jr., associate general counsel at PayPal, said while the site itself is not in violation of its Acceptable Use Policies (AUP), there have been cases where PayPal has identified accounts selling goods that violate its policy and in those cases, the company has exited those merchants from its system. 

“PayPal proactively monitors sellers with PayPal accounts who use the Paylvy platform to ensure the products they are selling are in compliance with our AUP, and we take appropriate action when violations are discovered,” Christin said.

The proprietor of PayIvy (quite possibly this guy, according to many of his fellow Hackforums users) makes money off of the service by selling “premium” accounts, which apparently offer repeat sellers a way to better track and manage their sales. Appropriately enough, among his ebook offerings via PayIvy is a tutorial on how to avoid getting one’s account banned or limited by PayPal. PayIvy did not respond to requests for comment.

Sh1eld makes clear how he feels about his users selling hacked accounts to pay services via his site in this thread, where he posts about takedown requests from a company representing Netflix.

“We are not under any obligation to follow any site’s TOS [terms of service],” he wrote. “However, we will take actions regarding copyrighted content, malicious files, or child pornography.”

I wonder how this individual would feel about people selling stolen PayIvy premium accounts?

If you’re curious about the underground’s interest in and valuation of your online accounts, take a look at my primers on this subject, including The Value of a Hacked Email Account and the Value of a Hacked PC. Want pointers on how to avoid becoming the next victim? Check out my Tools for a Safer PC tutorial.

Update, 10:33 a.m. ET: PayIvy just sent the following message to all of its sellers: “Starting May 15th, PayIvy will be banning all netflix accounts. If you are still selling these accounts, we advice you to stop as your paypal account will be limited as part of PayPal AUP. You have 9 days to delete your Netflix products before we do a search and remove them ourselves.”

For the second time in a year, nationwide beauty products chain Sally Beauty Holdings Inc. says it is investigating reports of unusual credit and debit card activity at some of its U.S. stores.

Last week, KrebsOnSecurity began hearing from multiple financial institutions about a pattern of fraudulent charges on cards that were all recently used at Sally Beauty locations in various states. Reached for comment on Sunday about the fraud pattern suggesting yet another card breach at the beauty products chain, Sally Beauty issued the following statement this morning:

“Sally Beauty Holdings, Inc. is currently investigating reports of unusual activity involving payment cards used at some of our U.S. Sally Beauty stores. Since learning of these reports, we have been working with law enforcement and our credit card processor and have launched a comprehensive investigation with the help of a leading third-party forensics expert to aggressively gather facts while working to ensure our customers are protected. Until this investigation is completed, it is difficult to determine with certainty the scope or nature of any potential incident, but we will continue to work vigilantly to address any potential issues that may affect our customers.”

Their statement continues: “Consistent with our ‘Love it or Return It’ policy, customer security and confidence remains our number one priority. As a result, we encourage any customer who is concerned about the security of their payment cards to call our Customer Service Hotline at 1-866-234-9442, so that we can assist them in addressing any potential concerns. Sally Beauty will, as appropriate, provide updates as we learn more from our investigation.”

In addition, the company also sent out an urgent alert today to its employees, asking associates to direct any customers with credit card issues to the Sally Beauty Web site or to call customer service. “We hadn’t gotten an email like that since last year when we had our breach,” the Sally Beauty employee said on condition of anonymity.

On March 5, 2014, this publication first reported that a batch of more than 282,000 cards that went up for sale on Rescator[dotc]cc — the same site that was first to sell cards stolen in the Home Depot and Target breaches — all traced back to customers who’d shopped at Sally Beauty locations nationwide. Asked about that pattern at the time, a company spokesperson said Sally Beauty had recently detected an intrusion into its network, but that neither its information technology experts nor an outside forensics firm could find evidence that customer card data had been stolen from the company’s systems.

But on March 17, 2014, Sally Beauty officially confirmed a breach of its network, but said its investigation determined that fewer than 25,000 card accounts were removed from its network. Nevertheless, a subsequent, exhaustive analysis of the Sally Beauty store ZIP codes listed in the cards for sale on Rescator’s site indicated that the 2014 breach impacted virtually all 2,600+ Sally Beauty locations nationwide.

Sally Beauty is not alone in dealing with separate card compromise incidents in a short period of time. Last month, hotel franchise management firm White Lodging disclosed that for the second time in a year, hackers had broken into point-of-sale systems at food and beverage outlets inside of many of its franchise locations.

It is possible that Sally Beauty locations are feeling the brunt of a large number of compromises at point-of-sale vendors, such as the recently announced breach among Harbortouch POS customers. However, at least two banks contacted by this author say the cards they were alerted to by Visa and MasterCard that correspond to the Harbortouch incident have very little overlap with the customer cards that were hit with fraudulent charges in the wake of their use at Sally Beauty locations recently.

Credit and debit card skimmers secretly attached to gas pumps are an increasingly common scourge throughout the United States. But the tables can be turned when these fraud devices are discovered, as evidenced by one California police department that has eschewed costly and time-consuming stakeouts in favor of affixing GPS tracking devices to the skimmers and then waiting for thieves to come collect their bounty.

One morning last year the Redlands, Calif. police department received a call about a skimming device that was found attached to a local gas pump. This wasn’t the first call of the day about such a discovery, but Redlands police didn’t exactly have time to stake out the compromised pumps. Instead, they attached a specially-made GPS tracking device to the pump skimmer.

At around 5 a.m. the next morning, a computer screen at the Redlands PD indicated that the compromised skimming device was on the move. The GPS device that the cops had hidden inside the skimmer was beaconing its location every six seconds, and the police were quickly able to determine that the skimmer was heading down a highway adjacent to the gas station and traveling at more than 50 MPH. Using handheld radios to pinpoint the exact location of the tracker, the police were able to locate the suspects, who were caught with several other devices implicating them in an organized crime ring.

This story in October 2014 the U.S. Justice Department‘s “COPS Office” indicates that the Redlands PD has taken the lead in using GPS technology to solve a variety of crimes, and had credited the technology with helping secure at least 139 arrests.

According to 3VR Inc., a San Francisco based surveillance and security firm, the Redlands PD has used the GPS technology to apprehend offender committing armed robberies, vehicle burglary, pharmaceutical burglary and robbery, cell store burglary and robbery, bike theft, laptop theft, constructions site theft, fire hydrant theft, metal theft, wire theft, 3rd row seat theft, cemetery theft, vending machine theft, mail theft, UPS parcel theft, residential burglary, tire theft, vehicle theft, cigarette theft, etc. “The technology has also been used to voluntarily track informants by sewing a unit into a purse,” 3VR wrote in a recent newsletter.

3VR notes that the GPS device used by the Redlands PD in the pump skimmer case runs for about six hours on a full battery, meaning cops have about six hours to locate the device before the GPS stops transmitting. However, the devices can be tweaked to extend the battery life, by allowing them to switch on only in the event the device actually is moved, and by decreasing the frequency with which the device beacons home.

One increasingly common type of gas pump skimmer — those equipped with Bluetooth technology — might not be as susceptible to these kinds of police tricks. Bluetooth skimmers are equipped to tap directly into the pump’s power supply, and to allow thieves to retrieve stolen card data wirelessly, just by pulling up to the compromised pump with a Bluetooth enabled laptop or smartphone and downloading the data without ever leaving the vehicle.

Unlike ATM skimmers, skimming devices attached to gas pumps usually are impossible for the average customer to spot because the skimmers are not stuck to the outside of the machine, but rather hidden inside after thieves gain access to the pump’s insides. I wouldn’t worry too much about pump skimmers, unless you’re accustomed to paying for fuel with a debit card: Having your checking account emptied of cash while your bank sorts out the situation can be a huge hassle and create secondary problems (bounced checks, for instance). Use a credit card instead.

How common are pump skimmers?  Thieves tend to attack multiple filling stations along a major interstate, as detailed in this July 2010 story about pump skimmer scammers. More recently, a law enforcement sweep of 6,100 gas stations in Florida last month turned up skimmers at 81 locations.

If you’re as fascinated by ATM and pump skimmers as I am, check out the rest of my skimmer series, All About Skimmers.

Last week, Allentown, Pa. based point-of-sale (POS) maker Harbortouch disclosed that a breach involving “a small number” of its restaurant and bar customers were impacted by malicious software that allowed thieves to siphon customer card data from affected merchants. KrebsOnSecurity has recently heard from a major U.S. card issuer that says the company is radically downplaying the scope of the breach, and that the compromise appears to have impacted more than 4,200 Harbortouch customers nationwide.

In the weeks leading up to the Harbortouch disclosure, many sources in the financial industry speculated that there was possibly a breach at a credit card processing company. This suspicion usually arises whenever banks start feeling a great deal of card fraud pain that they can’t easily trace back to one specific merchant (for more on why POS vendor breaches are difficult to pin down, check out this post.

Some banks were so anxious about the unexplained fraud spikes as stolen cards were used to buy goods at big box stores that they instituted dramatic changes to the way they processed debit card transactions. Glastonbury, Ct. based United Bank recently included a red-backgrounded notice conspicuously at the top of their home page stating: “In an effort to protect our customers after learning of a spike in fraudulent transactions in grocery stores as well as similar stores such as WalMart and Target, we have instituted a block in which customers will now be required to select ‘Debit’ and enter their ‘PIN’ for transactions at these stores when using their United Bank debit card.”

In a statement released last week to KrebsOnSecurity, Harbortouch said it has “identified and contained an incident that affected a small percentage of our merchants.”

“The incident involved the installation of malware on certain point of sale (POS) systems,” Harbortouch said in a written statement. “The advanced malware was designed to avoid detection by the antivirus program running on the POS System. Within hours of detecting the incident, Harbortouch identified and removed the malware from affected systems. We have engaged Mandiant, a leading forensic investigator, to assist in our ongoing investigation.”

The company said the incident did not affect Harbortouch’s own network, nor was it the result of any vulnerability in the PA-DSS validated POS software.

“Harbortouch does not directly process or store cardholder data,” the company explained. “It is important to note that only a small percentage of our merchants were affected and over a relatively short period of time. We are working with the appropriate parties to notify the card issuing banks that were potentially impacted. Those banks can then conduct heightened monitoring of transactions to detect and prevent unauthorized charges. We are also coordinating our efforts with law enforcement to assist them in their investigation.”

However, according to sources at a top 10 card-issuing bank here in the United States that shared voluminous fraud data with this author on condition of anonymity, the breach extends to at least 4,200 stores that run Harbortouch’s point-of-sale software.

Reached for comment about this claim, Harbortouch reiterated that the malware incident impacted a small percentage of its merchants.

“It was nowhere near all of our customers, that is simply a false statement” said Nate Hirshberg, marketing director at Harbortouch, declining to answer questions about how many locations the company serves. “This malware incident impacted individual merchant locations, not Harbortouch. Harbortouch is not a processing platform, not a gateway and we do not store any cardholder data. This is not an ongoing incident and the malware was eliminated rapidly upon detection.”

One thing is for sure: POS providers — and their myriad customers — have a massive target on their backs, and there are almost certainly many other POS companies that are dealing with similar problems. Stay tuned for further updates.