Hacking Guatemala’s DNS – Spying on Active Directory Users By Exploiting a TLD Misconfiguration

Guatemala City, By Rigostar (Own work) [CC BY-SA 3.0], via Wikimedia Commons.

UPDATE: Guatemala has now patched this issue after I reached out to their DNS administrator (and with a super quick turnaround as well!)

In search of new interesting high-impact DNS vulnerabilities I decided to take a look at the various top-level domains (TLDs) and analyze their configurations for errors. Upon some initial searching it turns out there is a nice open source service which helps DNS administrators scan their domains for misconfigurations called DNSCheck written by The Internet Foundation in Sweden. This tool helps highlight all sorts of odd DNS misconfigurations such as having an authoritative nameserver list mismatch between a domain’s authoritative nameservers and the nameservers set at the parent TLD nameservers (this was the issue causing the domain takeover vulnerabilities in iom.int and sen.gov covered in a previous post). While it doesn’t explicitly point out vulnerabilities in these domains it does let us know if something is misconfigured which is as good a place as any to start.

As of the time of this writing there are ~1,519 top level domains. Instead of scanning each TLD one by one by hand I opted to write a script to automate scanning all the TLDs. The output of the results of this script can be found under my Github repo titled TLD Health Report. The script was designed to highlight which TLDs have the most serious misconfigurations. After spending an hour or two reviewing the TLDs with configuration errors I came across an interesting issue with the .gt TLD. The error can be seen on this Github page and is as follows:

Soa Scan

Errors:

• No address found for maestro.gt. No IPv4 or IPv6 address was found for the host name.

Warnings:

• Error while checking SOA MNAME for gt. (maestro.gt). The SOA MNAME is not a valid host name.

Notices:

• SOA MNAME for gt. (maestro.gt) not listed as NS. The name server listed as the SOA MNAME is not listed as a name server.

…trimmed for brevity…

As can be seen from the above, multiple errors and warnings state that the domain name maestro.gt is not routable and is an invalid primary master nameserver (MNAME) for the .gt TLD. We can see this configuration in action by doing the following dig command:

mandatory@script-srchttpsyvgscript ~/P/gtmaster> dig SOA randomdomain.gt

; <<>> DiG 9.8.3-P1 <<>> SOA randomdomain.gt
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1444
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;randomdomain.gt.        IN    SOA

;; AUTHORITY SECTION:
gt.            0    IN    SOA    maestro.gt. alejandra.reynoso.cctld.gt. 2017012900 3600 600 604800 14400

;; Query time: 43 msec
;; SERVER: 172.16.0.1#53(172.16.0.1)
;; WHEN: Sun Jan 29 16:11:57 2017
;; MSG SIZE  rcvd: 101

As can be seen above, we receive an NXDOMAIN error (because the domain we requested does not exist) and along with it we get a statement of authority (SOA) record with maestro.gt listed as the master nameserver. Any requests for a non-existent domains under the .gt TLD will return a SOA record with this domain in it.

“Unresolvable” is Just Another Word for “Not Resolvable Yet“

If you’ve read some of the previous posts on DNS vulnerabilities that I’ve written you may have already guessed why this domain isn’t resolving. It’s because it doesn’t exist! As it turns out, it’s also open for registration. With a bit of searching and Google Translate I found that it was available for registration from the “Registro de Dominios .gt.” for around 60$ a year (with a minimum of two years registration).

Would you believe they take Visa?

While pricey, the registration was quick and completely automated which is more than I can say for many other TLDs in Central America. The only drawback was that my debit card was immediately frozen for questionable activity and I had to go online to unfreeze it so that the order could go through:

Apparently buying a Guatemalan domain name at 1:30 AM is grounds for your card being frozen.

Listening for Noise

Due to the SOA MNAME being sort of an esoteric feature of DNS I wasn’t sure what to expect in terms of DNS traffic. It was unclear from casual Googling what types of DNS clients and services would actually make use of this field for anything at all. For this reason I configured my Bind DNS server to answer all A record queries for any domain under .gt with the IP address of my server as a general catch-all. I set up tcpdump to dump all DNS traffic into a pcap file for later analysis and let the box sit overnight.

What I found in the morning was far from expected, when I checked the pcap data I found that computers from all over the country of Guatemala were sending DNS queries to my server:What in the world is going on? Looking further into the results only appears to yield more confusion:

A majority of the traffic was in DNS UPDATE queries from internal network IP addresses. The above is an example DNS UPDATE request attempting to update the zone of my DNS server to include an SRV record for an internal LDAP server belonging to a random government network in Guatemala.

So, why all of the DNS queries? If you’re familiar with what _ldap._tcp.dc._msdcs is for then you may have already spotted the forest from the trees.

Understanding Windows Active Directory Domains

To understand why we are getting this traffic we have to take a quick diversion to learn about Windows Active Directory (AD) Domains and DDNS (Dynamic DNS).

Windows domains are structured in much the same way as Internet domain names are structured. When creating a Windows domain you must specify a DNS suffix. This DNS suffix is comparable to a base-domain, with all of the sub-domains being either child domains or simply hostnames of machines under the parent domain. As an example, say we have a Windows domain of thehackerblog.com and we have a two Windows Active Directory (AD) networks under this parent domain: east and west. In this structure we would set up two domains under the parent domain of thehackerblog.com as east.thehackerblog.com and west.thehackerblog.com. If we had a dedicated web-server for internal company documentation for the east office under this domain it might have a hostname of documentation.east.thehackerblog.com. In this way we build up a domain-tree of our network, much how is done with the Internet’s DNS. The following is our example network as a tree:

These hostnames like documentation.east.thehackerblog.com allow computers and people to easily find each other in this network. The network’s DNS server will take these human-readable hostnames and turn them into IP addresses of the specific machines so that your computer knows where to request the information you’re looking for. So when, for example, you type in the internal website URL of documentation.east.thehackerblog.com into your web browser, your computer asks the network’s DNS server for the IP address of this hostname in order to find it and route traffic to it.

Given this structure a problem does arise. Since machine IP addresses can often change on these networks, what happens when the server behind documentation.east.thehackerblog.com suddenly has a change in IP (say due to a new DHCP lease, etc)? How can the network’s DNS servers figure out the new IP address of this machine?

Understanding Windows Dynamic Update

Window’s Active Directory networks solve this problem by the use of something called Dynamic Update. Dynamic update at a core is actually very simple. In order to keep records such as documentation.east.thehackerblog.com always fresh and accurate, occasionally these servers will reach out the network’s DNS server and update their specific DNS records. For example, if the documentation.east.thehackerblog.com IP address suddenly changed, this server would reach out to the domain’s DNS server and say “Hey! My IP changed, please update the documentation.east.thehackerblog.com record to point to my new IP so people know where I am!”. A real world comparable situation would be a person moving apartments and having to notify their friends, workplace, and online shopping outlets of this new location to reach them at via mail.

The process for this a bit technical (I’ve summarized it below) and is as follows (quoted from “Understanding Dynamic Update“):

• The DNS Client service sends a start of authority (SOA)–type query using the DNS domain name of the computer.
  • The DNS Client service sends a start of authority (SOA)–type query using the DNS domain name of the computer.
• The client computer uses the currently configured FQDN of the computer (such as newhost.tailspintoys.com) as the name that is specified in this query.
  • For standard primary zones, the primary server (owner) that is returned in the SOA query response is fixed and static. It always matches the exact DNS name as it appears in the SOA resource record that is stored with the zone. If, however, the zone being updated is directory integrated, any DNS server that is loading the zone can respond and dynamically insert its own name as the primary server (owner) of the zone in the SOA query response.
• The DNS Client service then attempts to contact the primary DNS server.
  • The client processes the SOA query response for its name to determine the IP address of the DNS server that is authorized as the primary server for accepting its name. It then proceeds to perform the following sequence of steps as needed to contact and dynamically update its primary server

    • It sends a dynamic update request to the primary server that is determined in the SOA query response.

      If the update succeeds, no further action is taken.

    • If this update fails, the client next sends a name server (NS)–type query for the zone name that is specified in the SOA record.
    • When it receives a response to this query, it sends an SOA query to the first DNS server that is listed in the response.

    • After the SOA query is resolved, the client sends a dynamic update to the server that is specified in the returned SOA record.

      If the update succeeds, no further action is taken.

    • If this update fails, the client repeats the SOA query process by sending to the next DNS server that is listed in the response.

• After the primary server that can perform the update is contacted, the client sends the update request and the server processes it.

  The contents of the update request include instructions to add host (A) (and possibly pointer (PTR)) resource records for newhost.tailspintoys.com and to remove these same record types for oldhost.tailspintoys.com, the name that was registered previously.

  The server also checks to ensure that updates are permitted for the client request. For standard primary zones, dynamic updates are not secured; therefore, any client attempt to update succeeds. For AD DS-integrated zones, updates are secured and performed using directory-based security settings.

The above process is long-winded and can be a lot to consume at first so I’ve bolded the important sections. Summing this up the client does the following:

• A SOA query with the full hostname of the current machine is sent to the network DNS server by the host. In our previous example this would be a query containing documentation.east.thehackerblog.com as a hostname.
• The network’s DNS server receives this request and replies with a SOA record of its own which specifies what the hostname of the master nameserver is. This may be something like ns1.thehackerblog.com, for example.
• The client then parses this SOA record and then reaches out to ns1.thehackerblog.com specified in the previous server’s SOA MNAME reply with the DNS update/change it would like to make.
• The ns1.thehackerblog.com DNS server receives this update and changes its records to match accordingly. The DNS record for documentation.east.thehackerblog.com now points to the host’s new IP address and everything routes properly again!

Windows Domain Naming Conventions & Best Practice Guidelines

The final piece of information we need to absorb before we understand our Guatemalan DNS anomaly is the following requirement for Windows domains:

“By default, the DNS client does not attempt dynamic update of top-level domain (TLD) zones. Any zone that is named with a single-label name is considered to be a TLD zone, for example, com, edu, blank, my-company. To configure a DNS client to allow the dynamic update of TLD zones, you can use the Update Top Level Domain Zones policy setting or you can modify the registry.”

So by default you can’t have a Windows domain of companydomain, for example, because it’s single-label and is considered by Windows to be a top level domain (TLD) like .com, .net, or .gt. While this setting can (hopefully) be disabled on each individual DNS client (read: every computer in your network), it’s much easier just to have a domain of companydomain.com. While not all companies follow this direction, it is highly recommended because having a single-label domain could potentially cause many conflictions down the line if your internal network domain ends up being registered as a top-level domain in the Internet’s DNS root zone (an issue known as Name Collision which was, and continues to be, a hotly discussed topic in Internet discussions).

Putting it All Together – Becoming Master of Your Domain!

Now that we have all of this information, the situation becomes quite a bit more clear. The country of Guatemala like has many Windows networks which have internal Windows domains that are sub-domains of the .gt TLD. For example, if Microsoft had an office in Guatemala their internal network domain might be something like microsoftinternal.com.gt. _These machines periodically make dynamic DNS updates to their network DNS server to ensure that all of the network’s hostnames are up-to-date and routing properly. For a variety of reasons such as prevention of name collisions with external company Internet sub-domains, it’s also likely that many networks make use of internal network domains which don’t exist on the wider Internet. For example in this case Microsoft may use an internal active directory domain name such as _microsoftinternal.com.gt since hosts under this domain wouldn’t ever have naming conflicts with external Internet sub-domains for microsoft.com.gt.

The issue is, many subtle DNS misconfigurations can cause DNS requests which should be for the internal domain’s DNS server, to hit the top level domain’s (TLD) servers instead. This issue is known as name collision and is a well-documented issue which is sometimes referred to as DNS queries “leaking”. If a machine uses a DNS resolver which is not aware of the internal network’s DNS setup, or is misconfigured to pass these queries on to the outer-Internet’s DNS servers then these queries are leaked to the .gt TLD servers since they are the delegated nameservers for that entire TLD space. The problem of name collisions is incredibly common and is the same reason why new TLDs are often put up to a large amount of scrutiny to make sure that they aren’t a common internal network domain name such as .local, .localhost, .localdomain, etc.

When this leaking occurs for an internal domain that is part of the .gt extension, the DNS UPDATE query sent by a machine attempting to update an internal DNS record accidentally is sent to the .gt TLD servers. Say, for example, we are a host on the microsoftinternal.com.gt internal domain and we send our DNS UPDATE query to a com.gt TLD server. The following is an example of this query performed with the nsupdate DNS command line tool (available on most OS X and Linux systems):

mandatory@script-srchttpsyvgscript ~/P/gtmaster> nsupdate -d
> prereq nxdomain printer.microsoftinternal.com.gt
> update add printer.microsoftinternal.com.gt 800 A 172.16.3.131
> send
Reply from SOA query:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id:  25175
;; flags: qr rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;printer.microsoftinternal.com.gt. IN    SOA

;; AUTHORITY SECTION:
com.gt.            0    IN    SOA    maestro.gt. alejandra.reynoso.cctld.gt. 2017012812 3600 600 604800 14400

Found zone name: com.gt
The master is: maestro.gt
Sending update to 52.15.155.128#53
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:  62645
;; flags:; ZONE: 1, PREREQ: 1, UPDATE: 1, ADDITIONAL: 0
;; PREREQUISITE SECTION:
printer.microsoftinternal.com.gt. 0 NONE ANY    

;; UPDATE SECTION:
printer.microsoftinternal.com.gt. 800 IN A    172.16.3.131


Reply from update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOTAUTH, id:  62645
;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;com.gt.                IN    SOA

> 

We can see that the query resulted in a NXDOMAIN error but regardless an SOA response is returned with maestro.gt as the master nameserver. Since this is exactly what the host was looking (e.g. it’s a valid SOA response to this query) it parses out maestro.gt MNAME and proceeds to send the DNS update query to the server behind maestro.gt hosted at 52.15.155.128 (my server). This query ends in failure because I’ve set up my DNS server to reject all update requests from clients (since I don’t want to actively interfere with these machines/networks). This is why all of these random machines are sending DNS update requests to my maestro.gt server.

What we have is essentially an new DNS issue which is the result of name collisions leading to dynamic DNS updates being sent to whatever server is listed under a TLD’s SOA record. Unlike regular name collisions though, we don’t need to register an entirely new TLD to exploit this issue we just need to register the domain name listed under an existing TLD’s SOA record (as we did with maestro.gt).

Try It Yourself – A Quick Lab

Note: Feel free to skip this section if you’re not interested in replicating these results personally.

If you’d like to see this behavior in action it’s actually fairly simple to replicate. We’ll use Windows 7 as our example OS inside of VMware along with Wireshark running to follow all the DNS traffic being sent by our machine.

To change your computer’s DNS suffix to match what a Guatemalan active directory domain might look like, go to Start > Computer > System Properties > Change Settings > Change > More and set the primary DNS suffix to any domain under the .gt TLD:

Click OK and apply this new setting, you will be prompted to restart your computer. Go ahead and do so and keep Wireshark running while Windows is starting up. You should see that during the startup process a few Dynamic Update queries are issued:

The above screenshot of my Wireshark output shows that the Windows VM did the following upon startup:

• Attempts to query the 8.8.8.8 DNS resolver for the SOA of mandatory.dnsleaktestnetwork.com.gt (Resulting in an NXDOMAIN error because that record doesn’t exist on the public Internet which 8.8.8.8 is configured to query).
• Windows then parses this NXDOMAIN response and finds a valid SOA MNAME record of maestro.gt in the response sent from the TLD.
• Windows then reaches out to maestro.gt (my server) and attempts to do a DNS UPDATE request. This fails because my server is configured to refuse all of these requests.
• Windows then attempts the entire process over again, this time in all capital letters instead (???).

We’ve now seen in-action exactly what goes on with these Windows machines that are hitting the maestro.gt server.

So, Just How Common Is This Issue?

Since Guatemala, and by proxy, the .gt domain space, is relatively small in comparison to the bigger top level domains such as .com, .net, .org, .us, etc we can only see a small fraction of the global number of DNS requests made due to this issue. On the larger scale, it’s likely that a huge amount of traffic is generated to many of the more-popular TLD nameservers from networks which utilize this TLD in their AD domain.

Luckily, while I haven’t been able to find any previous exploit research on Dynamic Update SOA hijacking specifically, we do have a great deal of past-research on name collisions in general. This is largely because of worries about newly issued TLDs conflicting with internal network domains which are utilized by computers/networks across the world. For a good overview of this problem and research into how widespread it is I recommend reading Name Collision in the DNS, a report commission by ICANN to better understand the name collision problem and its effects on the wider Internet. This reports contains lots of real-world data on just how big the name collision problem is and allows us a good base to extrapolate how widespread this specific SOA Dynamic Update variant is in the wild.

Perhaps the most damning piece of real world data that was collected and presented in this report is the following table:

The above table is from a ranked list of requested TLDs ordered by number of DNS requests to the roots (in thousands). The list in the report contains the top 100 TLDs in the original report but the above image truncates it to just 20. This table sheds some light on the raw number of leaked DNS requests actually hitting the roots that were never meant to end up there at all. As can be seen, requests for the invalid TLD of .local account for more requests to the DNS roots than all .org and .arpa requests combined – ranking as the 3rd most requested TLD overall! The following pie chart also sheds further light onto the scale of the problem:

The above chart shows that a shocking 45% of requests to the root servers are for TLDs which do not exist and are likely the result of leaked DNS requests. Perhaps even more shocking is that the number is likely even higher due to the possibility of internal domains having a valid top level domain but an invalid second level domain, such as the previously mentioned microsoftinternal.com.gt. These requests would appear valid to the roots (since .gt is a valid record in the root zone) but would be revealed as leaked/invalid once they hit the com.gt TLD servers.

What’s also very interesting is that the report appears to mention that they observed traffic stemming from this problem but did not appear to investigate it much:

5.4.5 Name includes _ldap or _kerberos at the lowest level

Patterns observed show many requests of one of the forms:

• _ldap._tcp.dc._msdcs.
• _ldap._tcp.pdc._msdcs.
• _ldap._tcp..\_sites.dc.\_msdcs.
• _ldap._tcp..\_sites.gc.\_msdcs.
• _ldap._tcp.._sites.
• _ldap._tcp.
• _kerberos._tcp.dc._msdcs.
• _kerberos._tcp..\_sites.dc.\_msdcs.
• _kerberos._tcp.._sites.
• _kerberos._tcp.

Typically these are queries for SRV records, although sometimes they are requests for SOA

records. They appear to be related to Microsoft Active Directory services.

Look familiar? The above remark about SOA records is almost certainly traffic being generated from this exact same issue. It is mentioned again later in the report as well:

Although the overwhelming majority of the observed DNS traffic in this study has been name lookups (i.e., conventional queries), some other DNS requests have been detected. These are mostly Dynamic Updates—traffic that should never reach the root servers. After all, changes to the DNS root are not carried out using Dynamic Updates, let alone ones originating from random IP addresses on the Internet. The most likely explanation for this behavior would be misconfigured Active Directory installations that leak from an internal network to the public Internet. It should be noted however that this study did not examine the root cause of this traffic.

The statement eventually concludes with:

It is not known what impact, if any, this might have on the end client or how this could change its subsequent behavior. Currently, no significant harm appears to result when these Dynamic Update requests receive a REFUSED or NOTAUTH error response. Returning a referral response after example is delegated at the root might cause these clients to fail or behave unexpectedly. It seems likely that this would create end user confusion and support problems for the organization’s IT department.

Sadly this report doesn’t make any statements about the actual volume of this traffic so we’re unable to get a good idea of the percentage/total scale of this issue in comparison to other name collisions. The report also fails to realize that the security considerations are actually much larger in scope due to the fact that a SOA MNAME returned by TLDs could potentially be a completely arbitrary domain (as was the case with .gt and maestro.gt). This allows for potentially any attacker to get in the middle of these DNS transactions via a domain registration just as we exploited for .gt.

Given this admittedly limited information, I don’t think it’s a big reach to assume that this issue is actively occurring at-scale and many of the top level domain servers must see a non-trivial amount of traffic from this problem. Depending on the specific configuration of each TLD’s SOA record they may also be vulnerability to exploitation if their SOA MNAME is available for registration like the case was here.

Fixing the Problem & Wider TLD Security Considerations

Given that the barrier to entry for exploiting this problem is so low (essentially just buying a domain name), and the impact seems to be of concern for a number of reasons, I don’t feel it would be unreasonable to require TLDs to ensure that the SOA MNAME field of their responses be non-resolvable, or resolvable to a TLD-controlled server which refuses to process these dynamic update queries. Given that this mistake appears to be easy to make due to the SOA MNAME field not really being used for DNS-slave replication much anymore, I don’t feel this change should be too unreasonable. Sadly, if you review over the results of the generated TLD Health Report mentioned earlier in this post you’ll see that a shocking number of TLDs are actually very broken in the functional sense. Given this fact, the hope for enforcement of this security concern is low given that even simple things like “having all the TLD nameserver in a working state” are not currently universally true for TLDs. If we can’t even ensure that TLDs are healthy in general how can we ever hope to ensure that all TLDs are mitigating this issue?

Calling on network administrators to prevent issues of this type in their own networks does not appear to be a winnable war either given the scale of DNS name collisions. While it’s important to notify and ensure that network admins understand this risk, ensuring that 100% of networks always configure things properly is a bit of a pipe dream.

A Privacy Nightmare

Given the fact that these machines in Guatemala appear to call out on a pretty regular basis there are some fairly big privacy concerns with this issue. Many of the DNS requests had hostnames which were very personal in nature (think NANCY-PC, MYLAPTOP, etc). Since a DNS UPDATE query is sent due to a variety of triggers this would mean the user’s personal computer would be phoning home to my nameserver every time they travelled, changed networks, or rebooted their machine. This would make it easy to monitor someone as they take their laptop from work, to home, to the local cafe, and so on. The internal IP addresses add to the privacy risk as well, given that it helps map out the remote network’s IP scheme. All the while these people have no idea that their computer was silently reporting their location. Imagine something like the following being built based off of this data:

Note: The above map doesn’t reflect any real data and is purely for illustrative purposes.

Needless to say, the privacy concerns might raise a few eyebrows.

Ethical Handling of Research Data

Out of respect for the privacy of the people of the Guatemala the raw update query data collected from this research will never be published and has been securely deleted. Additionally, all further traffic from these computers will be routed to 127.0.53.53 to ensure that no further UPDATE queries will be sent for the rest of my maestro.gt domain registration term (of about two years). The IP address 127.0.53.53 is special because it was specifically commissioned by ICANN to warn network administrators of name collision issues. The hope being that if a network admin/tech savvy user in Guatemala sees DNS traffic being routed to this IP they will Google it and immediately find that their network/AD setup is vulnerable to this issue.

Additionally I’ve emailed the DNS admin of the .gt TLD informing her of this issue and asked that she change the MNAME to something no-longer resolvable. Since nobody else can abuse this, combined with the fact that I’m sinkholing all DNS traffic to 127.0.53.53, combined further with the fact that my domain registration term is two years – I’ve decided to publish this post despite the MNAME still being maestro.gt as of this time_._ I imagine that two years of time for a patch falls within reason for patch expectations :).

—

Overall this was a fun experiment and highlights some very interesting ramifications of even minor TLD misconfigurations such as this one.

Until next time,

-mandatory