ASA5505 - Blocking internal traffic between 2 servers

[Richard Lawes]

Hi guys/ladies

I have a cisco ASA5505, it runs a wide site to site VPN network and has 4 servers connected to it

10.50.15.4 > fileserver

10.50.15.5 > domain controller (exchange)

10.50.15.6 > terminal server

10.50.15.7 > terminal server

Now yesterday i removed 10.50.15.6 and replaced it with a new terminal server with the same ip address, ever since the ASA is blocking traffic between it and the domain controller (example)

2

Oct 27 2012

14:51:05

106007

10.50.15.6

55978

DNS

Deny inbound UDP from 10.50.15.6/55978 to 10.50.15.5/53 due to DNS Query

What has me baffled is the only thing different between today and yesterday is the new server is windows server 2008 and the old one was windows server 2003. The new server has the same LAN ip address as the old one to make the changeover seamless for the users.

Any idea why all the sudden my ASA has decided to block the traffic between those machines? all the other machines can talk to it fine just not the domain controller, and seeing that this is a terminal server naturally you can see the problem i face!

Any help you can give would be great as this router has worked flawlessly for 2 years now without any config changes and i cant work out why its blocking traffic between those 2 machines.

[Richard Lawes]

Okay some more background,

If i remove the new terminal server from the domain example.local and its running locally no problems at all you can ping from the domain controller to the new terminal server, the minute i log the terminal server back into the local domain the ASA lights up and starts blocking everything again,

[Jennifer Halim]

It's blocking it due to DNS query. Does it resolve the DNS for the domain name? Which DNS server are you pointing it to?

If you temporarily put that domain controller domain name to resolve on the hosts file, does it still fail?

[Richard Lawes]

It's blocking icmp as well, it has stages where momentarily it starts letting it through then it starts blocking it again. The servers dns is set to 10.50.15.5 which is local to the machine as they are both in the same 255.255.255.0 subnet plugged into the 5505's switch module

[Jennifer Halim]

Strange.. can you pls share the config.

Also, try to reload the ASA.

[Marcel Tempelman]

Try disabling Threat detection. I've seen very odd behaviour because the ASA was reacting on false positives.

[Julio Carvajal]

Hello,

Try what jennifer and  mtempelman said and if that does not work share the configuration..We might need to take some captures.

Regards,

[Richard Lawes]

Hi guys,

First up thanks for the help, i had to leave for a flights

1: have rebooted ASA no change

2: created ACL permit rule for that ip for icmp,tcp,udp no change

3: turned off threat detection, no change

I also found a doc about networking services and windows scaling which was new on windows7/vista which has been said to be a known issue with older routers, so i turned of TCP offloading and disabled window scaling for TCP on the server but that hasnt seemed to help either.

Its just bizarre that the other 2 servers that arent domain controllers have no issue contacting the machine but the domain controller does. I think its some level of security function on the ASA and the new server sending DNS queries back to the domain controller is triggering it. I do notice after a while you can ping the new machine but as soon as you go to do pretty much anything on the new machine sure enough it fires off a DNS request and the ASA kicks in and starts blocking (ICMP immediately starts getting blocked too). This is really hinting as some kinda threat detection problem because it seems like the block kicks in due to how the machine is sending DNS but it does expire after some time.

[Julio Carvajal]

Hello Richard,

Do the following:

cap asp type asp-drop all circular-buffer

Then try to access that server, afterwards run the command and provide us the output

show cap asp | include 10.50.15.6

Also, why is the ASA receiving this DNS query if this traffic is on the same broadcast domain so traffic should not reach the ASA ( let me know if I am missing something on this one)

Also if you do a show shun what do you get?

[Richard Lawes]

Exactly, why is the ASA coming into the picture at all because its local network traffic!

I just rebooted the new server, and for about 5 minutes i could happily start logging in the users and setting them up then sure enough the ASA kicks in and starts blocking it again (naturally this is a killer as the domain controller is authenticating the users and when the ASA blocks the traffic everything stops).

I just added:

10.50.15.5 domain.local

10.50.15.5 org1.domain.local

10.50.15.4 org2.domain.local

To the hosts file to see if we can force local communication, although i dare say a DNS request sent to an outside location is going to trigger the block again but ill see how i go.

the servers network config:

ip: 10.50.15.6

sn: 255.255.255.0

gw: 10.50.15.254 (asa)

dns: 10.50.15.5

Now it has started working again but the question is for how long before the machine does something the ASA doesnt like. It would be nice if there was somewhere in my ASA that i can go, you know that ip 10.50.15.6 i dont care what it does dont touch its traffic

[Julio Carvajal]

Hello Richard,

Can you answer with the information I requested on the last post??

cap asp type asp-drop all circular-buffer

Then try to access that server, afterwards run the command and provide us the output

show cap asp | include 10.50.15.6

Also if you do a show shun what do you get?

I would like to see the configuration you have on this ASA,

Remember to rate all of the helpful posts

[Richard Lawes]

Result of the command: "show cap asp | include 10.50.15.6"

  15: 10:09:21.796849 802.1Q vlan#1 P0 10.50.15.6.58810 > 10.50.15.5.389:  udp 163

  16: 10:09:22.189153 802.1Q vlan#1 P0 10.50.15.6.58810 > 10.50.15.5.389:  udp 163 Drop-reason: (acl-drop) Flow is denied by configured rule

  17: 10:09:22.596252 802.1Q vlan#1 P0 10.50.15.6.50855 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

  18: 10:09:23.625913 802.1Q vlan#1 P0 10.50.15.6.50855 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

  19: 10:09:24.625227 802.1Q vlan#1 P0 10.50.15.6.50855 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

  20: 10:09:26.635236 802.1Q vlan#1 P0 10.50.15.6.50855 > 10.50.15.5.53:  udp 86

  25: 10:09:30.653500 802.1Q vlan#1 P0 10.50.15.6.50855 > 10.50.15.5.53:  udp 86

  27: 10:09:34.655025 802.1Q vlan#1 P0 10.50.15.6.137 > 10.50.15.255.137:  udp 50 Drop-reason: (acl-drop) Flow is denied by configured rule

  28: 10:09:34.655071 802.1Q vlan#1 P0 10.50.15.6.138 > 10.50.15.255.138:  udp 237

  29: 10:09:34.655193 802.1Q vlan#1 P0 10.50.15.6.138 > 10.50.15.5.138:  udp 237 Drop-reason: (acl-drop) Flow is denied by configured rule

  30: 10:09:34.764700 802.1Q vlan#1 P0 10.50.15.6.49854 > 10.50.15.5.88: S 1487640872:1487640872(0) win 8192 Drop-reason: (acl-drop) Flow is denied by configured rule

  31: 10:09:34.899337 802.1Q vlan#1 P0 10.50.15.6.56407 > 10.50.15.5.53:  udp 34 Drop-reason: (acl-drop) Flow is denied by configured rule

  32: 10:09:35.901946 802.1Q vlan#1 P0 10.50.15.6.56407 > 10.50.15.5.53:  udp 34 Drop-reason: (acl-drop) Flow is denied by configured rule

  33: 10:09:36.915937 802.1Q vlan#1 P0 10.50.15.6.56407 > 10.50.15.5.53:  udp 34 Drop-reason: (acl-drop) Flow is denied by configured rule

  34: 10:09:37.773916 802.1Q vlan#1 P0 10.50.15.6.49854 > 10.50.15.5.88: S 1487640872:1487640872(0) win 8192 Drop-reason: (acl-drop) Flow is denied by configured rule

  35: 10:09:38.942715 802.1Q vlan#1 P0 10.50.15.6.56407 > 10.50.15.5.53:  udp 34 Drop-reason: (acl-drop) Flow is denied by configured rule

  37: 10:09:42.937695 802.1Q vlan#1 P0 10.50.15.6.56407 > 10.50.15.5.53:  udp 34 Drop-reason: (acl-drop) Flow is denied by configured rule

  38: 10:09:43.788579 802.1Q vlan#1 P0 10.50.15.6.49854 > 10.50.15.5.88: S 1487640872:1487640872(0) win 8192 Drop-reason: (acl-drop) Flow is denied by configured rule

  41: 10:09:55.803608 802.1Q vlan#1 P0 10.50.15.6.54962 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

  42: 10:09:56.814166 802.1Q vlan#1 P0 10.50.15.6.54962 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

  43: 10:09:57.820804 802.1Q vlan#1 P0 10.50.15.6.54962 > 10.50.15.5.53:  udp 86 Drop-reason: (acl-drop) Flow is denied by configured rule

[Richard Lawes]

Im not so keen on posting the configuration for our core router on an online forum

[Julio Carvajal]

Hello Richard,

Yeah, I understand what you mean by posting this online, You could send it to me on a private message or then open a case with us at the Cisco TAC.

ACL drop, I mean that would be expected as that traffic is not intended to reach the ASA. Maybe this could be a proxy arp issue.

Again without the configuration it will be hard to help so you have the options right now

Regards,

[Richard Lawes]

I just emailed it to you