We received an alert from our internal monitoring system that a domain that we oversee for one of our clients had a configuration error causing the www subdomain to not resolve properly.  We immediately ran a DNS report to see what was amiss and noticed that, not only was the www hostname missing from the zone file, but that all the email was no longer being routed to Microsoft 365 (formerly Office 365) but was now being routed through two mail servers that were new to us: mx1-us1.ppe-hosted.com. [148.163.129.50] and mx2-us1.ppe-hosted.com. [67.231.154.163]

Was one of our clients hacked by ppe-hosted.com?

It looked like a domain hijack had been executed and someone was stealing emails and routing them through two mail servers at mx1-us1.ppe-hosted.com and mx2-us1.ppe-hosted.com instead of the usual MX servers used by Microsoft 365.   This is a big deal for any company, but this particular client is in the medical space therefore HIPAA compliance was immediately a concern as we can’t have prospective clients emailing an unknown third-party or interacting with a cloned site on an unauthorized server.

The first step to figure out how this happened was to determine what was currently happening and then work backwards from there.  Step one is to determine where the mail is being routed.

mx1-us1.ppe-hosted.com and mx1-us1.ppe-hosted.com

Quick research showed us that mx1-us1.ppe-hosted.com and mx2-us1.ppe-hosted.com are the hostnames for a mail server run by a company called Proofpoint out of California.  We geolocated 148.163.129.50 and 67.231.154.163 and confirmed these IP addresses are, in fact, located in California.

We felt a little bit better knowing the traffic was not being routed overseas, however plenty of malicious traffic originates in and terminates in the United States, so we were still on high alert.

Let’s talk to the client

We never want to call a client to tell them about a major problem and not have a solution ready, but, in this case, we needed answers, and we needed them fast.  So we called the client and hit them with a barrage of questions stabbing in the dark for any information.

• The name Proofpoint did not mean anything to them
• They do not have any invoices or credit card charges from a company called Proofpoint
• Office 365 is their email provider
• The client had not tried to migrate to a new email provider
• They did not sign up for any new email filtering service
• No new IT company had been hired
• The client’s email was still working properly
• And we are the only ones who make changes to their DNS zone file.

That’s not good.

What is Proofpoint, anyway?

Who is Proofpoint?

According to Wikipedia, “Proofpoint is an enterprise security company based in Sunnyvale, California that provides software as a service and products for inbound email security, outbound data loss prevention, social media, mobile devices, digital risk, email encryption, electronic discovery, and email archiving.”

Neat, but…

Is Proofpoint legitimate?

We called Klik.Solutions to see if they had heard of Proofpoint.  Fortunately, Neil, their Senior Solutions Architect, answered on the first ring, and was familiar with Proofpoint.

Neil indicated Proofpoint is a legitimate SaaS company that provides email spam and content filtering and that Proofpoint integrates with Microsoft 365.  At this point we are starting to breathe a little easier.  Neil added that Proofpoint is a premium service and that it is not something you accidentally implement and using Proofpoint requires a fair amount of configuration as evidenced by the extensive article on how to integrate  Proofpoint with Microsoft 365 (or Office 365)

Proofpoint is definitely not something you accidentally configure, but just to be sure we called the client again and they confirmed that they had not signed up for Proofpoint and were definitely not paying Proofpoint.

I never signed up for Proofpoint!

The client was adamant they had never signed up for Proofpoint, but did recall that Proofpoint sends them occasional notes about email that has been quarantined.

Neil told us that Proofpoint was a premium service,  so someone is footing the bill for Proofpoint if it’s a paid service.  Who touches this domain that might be paying for Proofpoint on behalf of our client?

The various players are:

• Registrar: GoDaddy
• DNS Host: GoDaddy
• Email Host: Microsoft 365 (allegedly)
• Web Host: Glimmernet Technologies

Another quick call to the client confirmed that there is no IT company, consultant, intern, or nephew making changes.  Nope, just GoDaddy, Microsoft, and Glimmernet.  And we know it was not Glimmernet.

We will start looking at billing for the registrar and then work our way down the chain.

From GoDaddy…

This bill reminded us that they didn’t buy Office 365 directly from Microsoft, but instead purchased Office 365 from GoDaddy.

Wait.  What’s GoDaddy’s “advanced email security?”  Is that Proofpoint?!?!?

A little more poking around leads us to this article:  https://www.godaddy.com/help/what-is-advanced-email-security-20148 where the third paragraph acknowledges:

Advanced Email Security, a partnership with Proofpoint, protects your information by encoding email contents during transit. This means that if a message is intercepted or ends up in the wrong hands, unauthorized parties can’t read it.

Mystery solved.

GoDaddy’s “advanced email security” is outsourced to Proofpoint.

Crisis adverted.  The client paid for Proofpoint by purchasing “advanced email security” from GoDaddy.  GoDaddy was able to automatically update the DNS zone file because they were the DNS host.

Updates to the DNS Zone File

We are not 100% sure exactly what changed in the file, but we presume it was a change to the MX record to support new functionality from Microsoft 365 and/or Proofpoint.  The change from GoDaddy specifically added a new stanza for the Proofpoint SPF record.  The same change also removed our SPF record for transactional email and deleted the A record for the www subdomain.

(The updates for the MX and SPF records can be explained, but we have no idea why www was removed.)

Our investigation into Proofpoint was a red herring, but led us to an important lesson-learned that we’ll get to in a minute.  At least we can fix this issue and get the site back up quickly.

The rough timeline of events:

1. GoDaddy updated the DNS Zone file on June 30, 2020 as evidenced by the timestamp in the GoDaddy DNS manager.
2. On July 19, 2020 the Let’s Encrypt SSL certificate protecting the so-called “naked” domain and the www subdomain failed to automatically renew.
3. The Let’s Encrypt certificate would not renew because the www subdomain was no longer pointed to our server.
4. We identified a missing A record for the www subdomain
5. After restoring the A record we were immediately able to renew the SSL certificate.  (We actually revoked and reissued the certificate.)
6. We then restored the missing SPF and DKIM records for our transactional email service

After these changes everything went back to normal and Proofpoint was functioning properly along side the transactional email service.

Postmortem

We learned four important lessons during this ordeal:

1. GoDaddy outsources email security to Proofpoint (even if they do not make immediately clear that they are doing so)
2. Proofpoint is a reputable company
3. GoDaddy manages DNS records for Proofpoint services and will update the zone file as needed and…
4. Automatic updates to the DNS zone file may introduce new errors or undo custom settings

The last take-away is key.

Take proactive measures to monitor your DNS zone files for any changes by having a third-party service alert you to any changes to the zone file as they occur. We monitor for any unauthorized DNS changes as part of our WordPress Hosting Service, however, because website IP addresses did not change and the hostnames were not impacted, we did not get notified that DNS change was a ticking time bomb.  When the Let’s Encrypt certificate was due to be renewed was the first time we knew of an issue and the moment that the domain verification failed was when the site went down and all of the alarm bells went off.

It is a good idea to have a third-party service monitor your DNS zone files for ALL changes and make sure you get alerts in real time for any and all changes as the smallest changes that seem insignificant may become very important in the run.  It is very easy to misconfigure DNS.

One last note.  We want to give thanks to Neil at Klik.Solutions for adding a bit of sanity to this and calming our fears.  If you need managed IT services, managed infrastructure, or disaster recovery, Klik.Solutions should be your go-to resource.  Klik saved us a ton of time today making sure we didn’t go down the rabbit hole.