The Orphaned Internet – Taking Over 120K Domains via a DNS Vulnerability in AWS, Google Cloud, Rackspace and Digital Ocean

Recently, I found that Digital Ocean suffered from a security vulnerability in their domain import system which allowed for the takeover of 20K domain names. If you haven’t given that post a read I recommend doing so before going through this write up. Originally I had assumed that this issue was specific to Digital Ocean but this couldn’t be farther from the truth as I’ve now learned. It turns out this vulnerability affects just about every popular managed DNS provider on the web. If you run a managed DNS service, it likely affects you too.

The Managed DNS Vulnerability

The root of this vulnerability occurs when a managed DNS provider allows someone to add a domain to their account without any verification of ownership of the domain name itself. This is actually an incredibly common flow and is used in cloud services such as AWS, Google Cloud, Rackspace and of course, Digital Ocean. The issue occurs when a domain name is used with one of these cloud services and the zone is later deleted without also changing the domain’s nameservers. This means that the domain is still fully set up for use in the cloud service but has no account with a zone file to control it. In many cloud providers this means that anyone can create a DNS zone for that domain and take full control over the domain. This allows an attacker to take full control over the domain to set up a website, issue SSL/TLS certificates, host email, etc. Worse yet, after combining the results from the various providers affected by this problem over 120,000 domains were vulnerable (likely many more).

Detecting Vulnerable Domains via DNS

Detecting this vulnerability is a fairly interesting process, it can be enumerated via a simple DNS NS query run against the target’s nameservers. If the domain is vulnerable then the nameservers will return either a SERVFAIL or REFUSED DNS error. The following is an example query using the dig DNS tool:

ubuntu@ip-172-30-0-49:~/$ dig NS zz[REDACTED].net

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> NS zz[REDACTED].net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62335
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zz[REDACTED].net.                 IN      NS

;; Query time: 73 msec
;; SERVER: 172.30.0.2#53(172.30.0.2)
;; WHEN: Sat Sep 17 16:46:30 PDT 2016
;; MSG SIZE  rcvd: 42

The above response shows we’ve received a DNS SERVFAIL error indicating that this domain is vulnerable.

If we get a SERVFAIL response how are we supposed to know what the actual nameservers are for this domain are? Actually, dig has already found what nameservers the domain has but just hasn’t displayed them to us. DNS queries for a domain’s nameservers usually follow the following process:

• Query the DNS root nameservers for the list of nameservers belonging to the domain’s TLD (in this case, .net).
• Query one of the nameservers for the specified TLD of the domain for the nameservers of the domain.
• Query the returned nameservers for the domain for the nameservers for the domain (unclear why dig does this, considering you already know what they are from the nameservers from the .net nameservers).

*Note that many of these steps will be skipped if the results are already cached by your resolver.

The last step is what is causing dig to return this SERVFAIL error, we’ll skip it and just ask the nameservers for the .net TLD directly. First we’ll query what those are:

ubuntu@ip-172-30-0-49:~$ dig NS net.

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> NS net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 624
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;net.                           IN      NS

;; ANSWER SECTION:
net.                    2597    IN      NS      b.gtld-servers.net.
net.                    2597    IN      NS      c.gtld-servers.net.
net.                    2597    IN      NS      d.gtld-servers.net.
net.                    2597    IN      NS      e.gtld-servers.net.
net.                    2597    IN      NS      f.gtld-servers.net.
net.                    2597    IN      NS      g.gtld-servers.net.
net.                    2597    IN      NS      h.gtld-servers.net.
net.                    2597    IN      NS      i.gtld-servers.net.
net.                    2597    IN      NS      j.gtld-servers.net.
net.                    2597    IN      NS      k.gtld-servers.net.
net.                    2597    IN      NS      l.gtld-servers.net.
net.                    2597    IN      NS      m.gtld-servers.net.
net.                    2597    IN      NS      a.gtld-servers.net.

;; Query time: 7 msec
;; SERVER: 172.30.0.2#53(172.30.0.2)
;; WHEN: Sat Sep 17 16:53:54 PDT 2016
;; MSG SIZE  rcvd: 253

Now we can query one of these nameservers for the nameservers of our target domain:

ubuntu@ip-172-30-0-49:~$ dig NS zz[REDACTED].net @a.gtld-servers.net.

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> NS zz[REDACTED].net @a.gtld-servers.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3529
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zz[REDACTED].net.                 IN      NS

;; AUTHORITY SECTION:
zz[REDACTED].net.          172800  IN      NS      dns1.stabletransit.com.
zz[REDACTED].net.          172800  IN      NS      dns2.stabletransit.com.

;; ADDITIONAL SECTION:
dns1.stabletransit.com. 172800  IN      A       69.20.95.4
dns2.stabletransit.com. 172800  IN      A       65.61.188.4

;; Query time: 9 msec
;; SERVER: 192.5.6.30#53(192.5.6.30)
;; WHEN: Sat Sep 17 16:54:48 PDT 2016
;; MSG SIZE  rcvd: 129

Now we can see that the nameservers for this domain are dns1.stabletransit.com and dns2.stabletransit.com and can target this set of nameservers specifically.

In order to find a list of domains vulnerable to this issue I used my copies of the zone files for the .com and .net TLDs which are available via Verisign (you have to apply to get access). These zone files have a list of every .com, and .net domain name along with what nameservers they use. Using this data we can find all domains which are hosted by a specific cloud provider because their nameservers will be those of these cloud providers. Once we have a list for a specific provider we can use a small Python script to query each domain to probe for the SERVFAIL or REFUSED DNS errors. Finally, we then use the cloud management panel to see if we can add these domains to our account, confirming the vulnerability exists.

Google Cloud DNS (~2.5K Domains Affected, Patched)

Google’s Cloud offering includes a managed DNS service line which has an easy import process for new domains. Documentation for doing so can be found here. The general process for doing so is the following:

• Go to the DNS management panel of your Google Cloud account: https://console.cloud.google.com/networking/dns
• Click the “+ Create Zone” button.
• Create a new zone with any name and a “DNS Name” of the vulnerable domain.
• Click the “Create” button to create this new zone.
• Note the list of nameservers that have been returned to you, in this example I’ve received the following:
• Check if the nameservers match the target nameservers, if they don’t just delete the zone and try again.
• Once you’ve finally gotten a matching list of nameservers you now have full control of the DNS for that domain.

Disclosure & Remediation Timeline

• Sep 9, 2016: Reported issue to Google bug bounty program, provided a list of vulnerable domains as well.
• Sep 9, 2016: Report triaged by Google Security Team.
• Sep 9, 2016: Vulnerability confirmed, internal bug filed for the issue.
• Sep 13, 2016: Reward of $1,337 awarded, donation matching available if given to a charity.
• Sep 14, 2016: Requested the reward be given to the Tor Project (making the donation $2,674). Received a Tor hoodie/swag from the Tor folks – thanks guys/gals!

Amazon Web Services – Route53 (~54K Domains Affected, Multiple Mitigations Performed)

Amazon’s managed DNS service line is called Route53. They have a number of nameservers which they randomly return to you which are distributed across multiple domains and TLDs. Previously I thought this was to defend against this specific type of vulnerability, however, since they are vulnerable I believe that this was more done to ensure uptime in the case of a TLD experiencing DNS issues.

The domain process for Route53 is complicated by the fact that they have a wide range of nameservers which can be returned to you. However the following process allows you to take over a target domain in just a few minutes. In order to automate this process I wrote a small Python proof-of-concept script which would create and delete Route53 zones until the proper nameservers were returned. One unique property of this vulnerability for Route53 was that you could get just one of the target’s nameservers returned to you instead of all four in a set like other managed DNS providers. This turns out to be just fine from an exploit scenario because you can simply keep the zone with three incorrect nameservers and one correct nameserver and keep creating zones until you have a couple zones with just one target nameserver in each zone. Then you can just replicate your DNS records across these four zones to set DNS for the target.

The process for this is as follows:

• Use the AWS Route53 API to create a new zone for a target domain.
• Check the resulting nameservers that were returned for this zone, if any of the nameservers match the target’s nameservers then keep the zone and remove it from the list of targeted nameservers. The following is an example of the nameservers returned for a domain:

• If none are shared with the target nameserver set, delete the zone.
• Keep repeating this process until you have X number of zones which have all of the target’s vulnerable nameservers.
• Now just create the DNS record you’d like for the target domain across all zones.

The following is a redacted example of creating four zones for a target domain, each zone containing just one of the target’s nameservers:

Using this method we can reliably takeover any of these 54K domains.

Disclosure & Remediation Timeline

• Sep 5, 2016: Contacted AWS security using PGP describing the full scope of the issue and with an attached list of vulnerable domains.
• Sep 5, 2016: (less than an hour from first contact): Response from AWS stating they will investigate immediately and follow up as soon as they know more.
• Sep 5, 2016: Responded to the team with an apology for contact them on labor day and thanking them for the quick response time.
• Sep 7, 2016: Contacted by Zack from the AWS security team, requesting a call to talk more about the issue and understand a disclosure strategy.
• Sep 7, 2016: Responded confirming that Sep 8 works for a call to discuss.
• Sep 8, 2016: Call with AWS to discuss the vulnerability and plans to alert affected customers and remediation of the issue.
• Oct 7, 2016: Follow up call with someone from the Route53 team discussing Amazon’s remediation strategy and next steps. Their plan was three pronged in approach:
  • Raise awareness by updating existing Route53 documentation to explicitly mention that nameservers should be changed if a zone is deleted from Route53. This was already updated when I had the call with them.
  • Add a UI warning in the AWS control panel which notifies users of this issue upon a user attempting to delete a Route53 zone.
  • Reach out to affected customers.

All of the above steps were indeed taken by Amazon. You now get the following warning when you delete a zone in Route53:

If you are an AWS customer using Route53, be sure to read this documentation about the risks of not changing your domain nameservers after deleting a zone from your account.

Disclosure notes: Overall this team was awesome to do disclosure with, they were super helpful and cared deeply about getting a proper fix in place. The response in less than an hour on labor day (late at night too) was crazy to see, very impressed. Great job guys :), wish all disclosure were like this.

Rackspace (~44K Domains Affected, Won’t Fix)

Rackspace offers a Cloud DNS service which is included free with every Rackspace account. Unlike Google Cloud and AWS Route53 there are only two nameservers (dns1.stabletransit.com and dns2.stabletransit.com) for their cloud DNS offering so no complicated zone creation/deletion process is needed. All that needs to be done is to enumerate the vulnerable domains and add them to your account. The steps for this are the following:

• Under the Cloud DNS panel, click the “Create Domain” button and specify the vulnerable domain and a contact email and TTL.
• Now simple create whatever DNS records you’d like for the taken over domain.

This can be done for any of the 44K domain names to take them over. Rackspace does not appear to be interested in patching this (see below) so if you are a Rackspace customer please ensure you’re properly removing Rackspace’s nameservers from your domain after you move your domain.

Disclosure & Remediation Timeline

• Sep 9, 2016: Reported vulnerability to the Rackspace security team, included a list of vulnerable domains.

• Sep 12, 2016: Rackspace responds with the following:

  “Thank you for your submission. We have taken it under advisement and will contact you if we require any additional information. For the protection of our customers, Rackspace does not disclose, discuss, or confirm security issues until a full investigation has occurred and any necessary patches, fixes, or releases are available. Rackspace usually distributes security disclosures and notifications through blog posts and customer support portals.__Please do not post or share any information about a potential security vulnerability in any public setting until we have researched, responded to, and addressed the reported vulnerability and informed customers, if needed. Our products are complex, and reported security vulnerabilities will take time to investigate, address, and fix.__While we sincerely appreciate reports for vulnerabilities of all severity levels, we will include your name on our website at http://www.rackspace.com/information/legal/rsdp if you report a previously unknown vulnerability, which Rackspace has determined to be of a high or critical severity, or in cases where there has been continued research or other contributions made by the person.__Thanks and have a great day.”

• Sep 14, 2016: Due to the previous email seeming to state that they won’t confirm the issue until a full investigation occurs – I ask that they notify me when remediation has occurred so I can properly coordinate releasing the vulnerability information to the general public.
• Oct 7, 2016: Due to a lack of response from vendor, notified them that a 90-day responsible disclosure timeline would be observed with a disclosure occurring regardless of a patch.
• Nov 7, 2016: Pinged Rackspace (no response as of yet) notifying them that disclosure will occur in 30 days.

• Dec 5, 2016: Received the following response from Rackspace:

  “Thank you again for your work to raise public awareness of this DNS issue. We’ve been aware of the issue for quite a while. It can affect customers who don’t take standard security precautions when they migrate their domains, whether those customers are hosted at Rackspace or at other cloud providers.

   

  If a customer intends to migrate a domain to Rackspace or any other cloud provider, he or she needs to accomplish that migration before pointing a domain registrar at that provider’s name servers. Otherwise, any customer of that provider who has malicious intentions could conceivably add that domain to his or her account. That could allow the bad actor to effectively impersonate the domain owner, with power to set up a website and receive email for the hijacked domain.

   

  We share multiple articles about DNS both on the Rackspace Support Network (https://support.rackspace.com/) and in the Rackspace Community (https://community.rackspace.com/). Our Acceptable Use Policy (http://www.rackspace.com/information/legal/aup) prohibits activities like this. Our support teams also work directly with customers to help them take the simple steps necessary to secure their domains.

   

  We appreciate your raising public awareness of this issue and are always glad to work with you.

   

  Sincerely,

  The Rackspace DNS Team”

Disclosure notes: Responsible disclosures that affect a large number of vendors usually take a full 90-days because one vendor drags their feet until the very end. In this case it appears Rackspace is that vendor. Their policy, specifically the following: “Rackspace does not disclose, discuss, or confirm security issues until a full investigation has occurred and any necessary patches, fixes, or releases are available.” does not instill confidence that anything is being done to address reported security vulnerabilities. To not confirm or discuss a reported security vulnerability until after remediation is a very odd approach and it’s unclear why such a policy would ever be in place. One would hope that all of the initial reports they receive are 100% clear in explanation, since they cannot ask any questions until after remediation of said unclear vulnerability has occurred. Additionally, the final response seems to be to the effect of “we write many articles about DNS, also if a customer doesn’t properly remediate this issue, they will be vulnerable but it is against our AUP to exploit this issue”. So this puts extra importance on raising awareness for this issue, since Rackspace does not appear interested in issuing a fix for this issue.

DigitalOcean (~20K Domains Affected)

For the full write up on how this issue affected Digital Ocean, please see this post.

Remediation Recommendation

Different cloud providers took different approaches to this issue. My recommendation for remediation of this issue is fairly straightforward and is the following:

• User adds the domain to their account via the cloud management panel.
• At this time the cloud provider returns a random list of nameservers for the domain owners to point their domain to. For example the following is an example list:
  • a-nameserver-one.example.com
  • a-nameserver-two.example.com
  • a-nameserver-three.example.com
• The cloud provider now continually queries the TLD’s nameservers for the list of nameservers that the domain has been set to. Once the user has set their nameservers properly the cloud provider stores the list of nameservers and the domain in a database.
• For any future zones created for this domain the cloud provider will only return nameservers which do not match the stored list of nameservers. This means that in order to use a newly created zone the domain owner will have to set their domain’s nameservers to a new nameserver set, ensuring that only the domain owner can actually carry out this action.
• The cloud provider will continually query the TLD nameservers to see if the domain’s nameservers have changed to the new set and will store the results in a database as we did in step 3.

The above method does add a bit of friction to the process of re-creating a zone for a domain, but completely prevents this issue from occurring. Since the friction is only equivalent to the initial domain import process it doesn’t seem to be too unreasonable but it is possible that provider won’t want to inconvenience customers this way.

Usefulness to Attackers

The attack scenario for this vulnerability can be split into two separate groups, targeted and un-targeted. Depending on the end goal of an attacker, either one could be chosen.

Targeted Attack

In the targeted use case you have an attacker who wants to take over a specific domain or list of domains belonging to their victim. In this case an attacker would set up a script to continually perform NS queries against the nameservers of the domains that they are targeting. The script will detect if it has received an SERVFAIL or REFUSED DNS error and will immediately attempt to allocate the target’s nameservers upon detecting no zone exists for the domain. Many different mistakes on the domain owners part could cause the zone to be deleted such as the cloud provider deleting the zone due to lack of payment, or if the company is changing providers, etc.

Un-Targeted Attack

The more likely attack scenario, in my opinion, would be an attacker who is merely interested in clean domains to be used for malware and spam related campaigns. Since many threat intelligence services will rate a domain based off of its age, length of registration, and cost to register there is a big advantage to hijacking existing domains over registering new ones. In addition to the hijacked domains often having past history and a long age, they also have WHOIS information which points to real people unrelated to the person carrying out the attack. Now if an attacker launches a malware campaign using these domains, it will be harder to pinpoint who/what is carrying out the attack since the domains would all appear to be just regular domains with no observable pattern other than the fact that they all use cloud DNS. It’s an attacker’s dream, troublesome attribution and an endless number of names to use for malicious campaigns.

Conclusion

This vulnerability is a systemic issue which affects all major managed DNS providers. It is very likely that more providers are affected which are not mentioned here. All managed DNS providers are encouraged to check their own implementations for this issue and patch/notify customers as soon as possible.

Until next time,

-mandatory