Troubleshooting HA Proxy DNS Intercept feature

[David Damerjian]

troubleshooting HA Proxy DNS Intercept feature includes what commands to run to see if it is working, checking the configuration, shortcuts in testing

The HA Proxy DNS Intercept feature is designed to allow Mobile IP subscribers roaming through a Foreign Agent to be able to reach their home network’s DNS server instead of the roaming partner’s DNS server which may not be reachable from the subscribers home network. The architecture and configuration of how to do this on the Home Agent is explained in the appropriate chapter in the PDSN Administration Guide, and so this article is not written to re-explain that. Rather this article focuses on troubleshooting this feature when it appears to not be working.

When configured properly, the intercept list created willinclude redirect lines for all of the roaming partners for which you want DNSpackets redirected, and pass-thru lines for all of the roaming partners forwhich you want to pass through DNS packets to those partners’ DNS servers.

For the pass-thru cases, if there are any, make sure thatthe list is complete, otherwise when roaming through partners for which aredirection is not included, DNS packets will be dropped! This would be a casewhere you are worst off attempting to implement this feature than notimplementing it at all, b/c you would have broken DNS for such roamers where ithad already been working. To avoid such oversights, implement a catch-all ruleas follows: pass-thru 0.0.0.0/0 to allow all DNS packets that are not specifiedto be redirected to be passed through.

For the redirect cases, a complete test of the feature wouldbe to try to access the internet roaming through all partners for whichredirect has been specified. This requires resources such as devices andpersonnel located in areas where the devices will be forced to roam.

If redirect does not seem to be working, you should confirmwith your partners what DNS addresses they are assigning. Note that if yourpartner is also using Starent, then the DNS addresses can be found in thedefault subscriber template (“dns (primary | secondary) <address>”) ofthe source context, and NOT in any named subscriber template, because theusername (NAI) is NOT known at the time that the DNS address is passed to thesubscriber via PPP negotiation. Here is an example of the PPP DNS exchange thatwould take place on the FA during call setup:

INBOUND>>>>> 16:54:34:471 Eventid:25000(0)

PPPRx PDU (18)

IPCP18: Conf-Req(34), Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0

<<<<OUTBOUND 16:54:34:472 Eventid:25001(0)

PPPTx PDU (20)

IPCP20: Conf-Nak(34), Pri-DNS=2.2.2.2, Sec-DNS=1.1.1.1

INBOUND>>>>> 16:54:34:623 Eventid:25000(0)

PPPRx PDU (18)

IPCP18: Conf-Req(35), Pri-DNS=2.2.2.2, Sec-DNS=1.1.1.1

<<<<OUTBOUND 16:54:34:623 Eventid:25001(0)

PPPTx PDU (20)

IPCP20: Conf-Ack(35), Pri-DNS=2.2.2.2, Sec-DNS=1.1.1.1

Then you should setup a test call and run a monitorsubscriber trace for the mobile device in question. DNS packets are easy toidentify and they use port 53. You should see the DNS requests coming from theFA, and if working, you should see DNS responses being sent back to the mobile,and if so, it is likely working (unless they are being passed through andsuccessfully responded to, which is unlikely if you already expect in advancefor that not to work). Here is an example of a request to get the IP address ofwww.yahoo.com, and the response, where FA = 65.80.120.44, HA = 8.9.40.35,mobile device = 20.20.20.30, DNS server = 69.44.78.89, and returned address forwww.yahoo.com is 69.147.76.15, just to give an idea of what to look for.

INBOUND>>>>> 16:20:30:728 Eventid:27000(0)

MIP-TUNNEL(IPv4-IPv4) Rx PDU

65.80.120.44> 8.9.40.35: 20.20.20.30.2467 > 69.44.78.89.53:  [udp sum ok] 3+ A?www.yahoo.com. [|domain] (ttl 127, id 62019, len 59) (ttl 251, id 0, len 79)

<<<<OUTBOUND 16:20:30:731 Eventid:27001(0)

MIP-TUNNEL(IPv4-IPv4) Tx PDU

8.9.40.35> 65.80.120.44: 69.44.78.89.53 > 20.20.20.30.2467:  [udp sum ok] 3 q: A?www.yahoo.com. 3/2/2 www.yahoo.com. CNAME www.wa1.b.yahoo.com.,www.wa1.b.yahoo.com. CNAME www-real.wa1.b.yahoo.com., www-real.wa1.b.yahoo.com.A 69.147.76.15 ns: wa1.b.yahoo.com. NS yf2.yahoo.com., wa1.b.yahoo.com. NSyf1.yahoo.com. ar: yf1.yahoo.com. A 68.142.254.15, yf2.yahoo.com. A68.180.130.15 (162) (DF) (ttl 245, id 46647, len 190) (ttl 255, id 0, len 210)

If you only see packets coming from the FA without response,then confirm that the source address is what is expected, and if not, then youmay need a trace from the FA side to determine why.

Make sure that there is a redirect rule in the interceptlist that matches the destination DNS server address, which should already bethe case if configured properly.

Note that monitor subscriber trace will NOT display theredirect address (if it is redirected) in either direction. (The mon suboutput is generated before the packet's destination address is changed bythe system (going TO the DNS server) and before the packet's source address ischanged by the system (coming FROM the DNS server)).

In order to confirm redirects are occurring for specificsubscriber, run command “show subscriber full  …”. First note whether the proxylist has been applied to the user, as seen in the “Proxy DNS Intercept List”field. If not, then check the configuration to make sure the list name isapplied to the subscriber template. As DNS requests are attempted, observe thecounters “ipv4 proxy-dns redirect”, “ipv4 proxy-dns pass-thru”, and “ipv4proxy-dns drop” to see which, if any counter is incrementing for each request.As a sidenote, the output for fields “Primary DNS Address” and “Secondary DNSAddress” have NO relevance on show subscriber full on the HA when using theIntercept feature.

The meaning of each counter is self-explanatory: redirectmeans the packet was redirected according to a rule, pass-thru means the packetwas passed through untouched according to a rule, and dropped means it wasdropped because no rule existed. If the count does not add up to the total DNSpackets, then it is also possible for packets to have been dropped by otherAccess Control Lists applied to subscribers, as such rules are applied firstbefore the DNS proxy list. The active input/output acl fields will indicatewhich if any ACL lists are applied, and “ipv4 input/output acl drop” counterswill track the numbers dropped. You can then check the rules for such ACLs tosee if any matches would cause a drop to occur.

There are some useful statistics that can be displayed ifthe issue seems to be widespread and tied to all subscribers of a particulartype versus just a specific subscriber:

“show dns-proxy statistics”

This includes statistics for each intercept list if thereare more than one and you want a snapshot.

“show dns-proxy intercept-list <list name> statistics”

Shows counts for redirected, passed-through, dropped, for EACHRULE in the particular intercept list

“show dns-proxy intercept-list <list name> rule<rule> statistics”

For EACH RULE, this adds counters for the number of requestssent to the DNS redirect server(s), the number of responses received from theserver(s), and number of timeouts.

Also you should be aware that when a redirect is applied, bydefault, the source address of the DNS packet is also changed, from thesubscriber’s address to the address specified in the configuration in therespective destination context with command:

ip dns-proxy source-address <interface source address forredirected DNS packets>

This will be different than non-redirected DNS packets whichwill maintain their subscriber source address, and, this could lead tounexpected routing/firewall issues where re-directed packets never make it tothe DNS server or don’t make it back due to this changed source address. It ispossible to specify that the DNS packet source address NOT be changed using theappropriate subscriber template command “proxy-dns use-subscriber-address-as-source”,and this may be worth trying as a temporary (or permanent) solution.

Here are some additional hints that could save you some timesetting up a troubleshooting session:

1) If you do not have a mobile device handy for which youcan run tests roaming through the partner that you are testing, in the case ofa laptop, you could spoof the DNS address assigned to a non-roaming mobiledevice, which you should have plenty of for testing, by using the DOS nslookupcommand. At the > prompt, enter a URL that should be resolvable to confirmthat it is. Then enter command “server <DNS address>” where the addressspecified is the DNS address in the redirect list that you want to test.Finally, enter that same URL resolved a moment ago and note whether it isproperly redirected.

2) You should already know the ip addresses of the ForeignAgents of the roaming partner(s) you are trying to troubleshoot. You could getthe list of subscribers currently connected through the FA(s) with the command“show subscribers fa <FA address>. Choose one that has a short TIME-IDLEand simultaneously do a monitor subscriber and show subscriber full to see ifDNS is working. Some active users may not be actively doing DNS, so you mayneed to try a few until you find one that does, but subscribers surfing theinternet will be making DNS requests.

3) The monitor logging feature for a specific subscriberwill save DEBUG level logging for that subscriber’s session without havingadverse impact on the system (much more information than monitor subscriber).In global configuration mode, enable with “logging monitor username<username>”. Then in Exec mode, “logging active” to turn on logging tothe terminal, run the test and save the output displayed, and “no loggingactive” to stop it. Or, just “show logs” after the session is complete willdisplay all logs in the buffer. Depending on the level of debug logging turnedon for various facilities, the output may be mixed with other output you are notinterested in. You could temporarily disable the other logging for the durationof the test to get just the relevant logs, or you could provide the full logsto Starent to extract the relevant ones.

4) Ultimately if subscriber traces show packets redirectedbut no response, a capture trace on the egress interface defined for DNS (with“ip dns-proxy source-address <address>”) will eliminate the Home Agentfrom being the cause. Traces and logs from the DNS server and other spots inthe transport layer between the PDSN and DNS server may also assist introubleshooting.

5) To see if a particular DNS server is able to resolve DNSrequests from a particular context (the destination context is what you wouldbe interested if troubleshooting DNS issues), there is a config command thatallows specifying a DNS server for all DNS requests manually typed in exec modein a particular context. DNS requests can be made indirectly by attempting toping a web url instead of the actual ip address. For example, to configure,BOTH of the following are required:

ipdomain-lookup

ipname-servers <primary server ip address> <secondary server ipaddress>

Then to test, for example, note that www.purple.com has been resolved to an ip address:

[dest]HA-PDSN# ping www.purple.com

PING www.purple.com (153.104.63.227) 56(84) bytes of data.

At least you would know that the DNS server is not only reachable, but also responding with proper answers.

Imported from Starent Networks Knowledgebase Article # 10840

[mabrande]

Here are a couple of additional notes regarding the Proxy-DNS feature:

• We only track 25 outstanding dns queries max per subscriber. There can be a maximum of 4,000 outstanding DNS queries per session manager.  If either of these limits are reached, the request is dropped.
  
• Each session manager will use a single source port # in the format of 1700+sessmgr instance number.
  
• The primary is declared to be DOWN if  no response is received with T1 seconds and assumed to be back UP again after T2 seconds. T1 is 10 seconds, and 60 seconds for T2.
  
• Use the 'sessmgr' facility for advanced debugging.
  

  event 12967              

  "DNS server <%s> in vpn <%s> defined by rule number %lu changes to new state : <%s> , reason : %s"

  info

  event 12968

  "DNS pkt is redirected to server <%s>, in vpn <%lu>"

  info