Re: ASK THE EXPERTS - TROUBLESHOOTING ASA, PIX AND FWSM - Page 5

ciscomoderator · ‎01-03-2011

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to learn how to address and troubleshoot common problems with Adaptive Security Appliances, Private Internet Exchange and Firewall Service Modules with Kureli Sankar. Kureli is an engineer supporting Cisco's firewall team in Research Triangle Park, North Carolina. Her team supports the Cisco Adaptive Security Appliance, Firewall Services Module, Cisco Security Manager, the Content Security and Control module, and the Zone Based Firewall module in Cisco IOS Software.

Remember to use the rating system to let Kureli know if you have received an adequate response.

Kureli might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through January 14, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

Kureli Sankar · ‎01-11-2011

For telnetting from outside to inside you need static translation. I thought you were trying outbound connection from high to low security.

Try this pls.

conf t

no nat (inside) 1 0 0

no global (outside) 1 int

static (inside,outside) 10.0.0.0 10.0.0.0 net 255.255.255.252

static (inside,outside) 20.0.0.0 20.0.0.0 net 255.255.255.0

Give it a shot. Watch the logs again.

-Kureli

MataKiera · ‎01-11-2011

I need to do ssh and telnet for outside to the inside (I am managing the firewall with ssh from my server and would like to do the same with my router and switch - for now, later on the management will bw from the inside). But the regular traffic (DNS, SIP, H323, FTP, ....) must be from inside to the outside.

here are the previous logs for telnet from inside to the outside

Jan 12 2011 07:27:33 fw-s-svn : %ASA-6-305012: Teardown dynamic ICMP translation from inside:20.0.0.101/512 to outside:172.16.0.1/25036 duration 0:00:33

Jan 12 2011 07:27:33 fw-s-svn : %ASA-7-609002: Teardown local-host inside:20.0.0.101 duration 0:00:33

Jan 12 2011 07:27:46 fw-s-svn : %ASA-4-106100: access-list INSIDEACL permitted udp inside/20.0.0.101(55282) -> outside/172.16.0.251(53) hit-cnt 1 first hit [0x2fd140f5, 0x0]

Jan 12 2011 07:27:46 fw-s-svn : %ASA-7-609001: Built local-host inside:20.0.0.101

Jan 12 2011 07:27:46 fw-s-svn : %ASA-6-305011: Built dynamic UDP translation from inside:20.0.0.101/55282 to outside:172.16.0.1/54645

Jan 12 2011 07:27:46 fw-s-svn : %ASA-6-302015: Built outbound UDP connection 86562 for outside:172.16.0.251/53 (172.16.0.251/53) to inside:20.0.0.101/55282 (172.16.0.1/54645)

Jan 12 2011 07:27:46 fw-s-svn : %ASA-6-302016: Teardown UDP connection 86562 for outside:172.16.0.251/53 to inside:20.0.0.101/55282 duration 0:00:00 bytes 68

Jan 12 2011 07:28:16 fw-s-svn : %ASA-6-305012: Teardown dynamic UDP translation from inside:20.0.0.101/55282 to outside:172.16.0.1/54645 duration 0:00:30

Jan 12 2011 07:28:16 fw-s-svn : %ASA-7-609002: Teardown local-host inside:20.0.0.101 duration 0:00:30

It seems that if I try to ping from the client on the inside I reach the server but if I try to ping from the router or the switch the ping fails. Also if I try ping from the server to the client it fails

Kureli Sankar · ‎01-12-2011

Tomaz,

For any traffic going through the firewall we need to have the following:

1. translation

2. Route

3. Permission

Now, the topology is something like this: (We would appreciate if all our readers would provide us a simple text based topology like this one).

20.0.0.0/24--R1---(inside)ASA(outside)---(172.10.0.250)R2---172.16.0.0/24 (server)

|

lo (200.0.1.2)

lo on switch 200.0.1.3

Now, on the firewall you need the following:

static (inside,outside) 200.0.1.2 200.0.1.2 netmask 255.255.255.255

static (inside,outside) 200.0.1.3 200.0.1.3 netmask 255.255.255.255

Now the above covers translation:

You do have routes via ospf:

O 200.0.1.3 255.255.255.255 [110/12] via 10.0.0.2, 8:13:54, inside

O 200.0.1.2 255.255.255.255 [110/11] via 10.0.0.2, 8:13:54, inside

You have ip any any for now as permission.

What you need to verify is that if R1 and the switch on the inside have a route to 172.16.0.0/24 via the firewall's inside IP address.

So, now you should be able to ssh, telnet to these inside R1's loop back from the server on the outside. If not I need to see the logs when it fails.

-KS

MataKiera · ‎01-12-2011

I checked the ospf routes on my R1 and S1 the both have the route. I enterd the static routes and then tried to telnet from inside to outside and it was denied. Then I tried to ssh from outside server to inside router and it was denied. on the inside I cant get the DNS working (dns server is on the outside)

Jan 12 2011 15:24:00 fw-s-svn : %ASA-7-609001: Built local-host inside:20.0.0.101

Jan 12 2011 15:24:00 fw-s-svn : %ASA-6-302013: Built outbound TCP connection 88236 for outside:172.16.0.251/513 (172.16.0.251/513) to inside:20.0.0.101/1023 (20.0.0.101/1023)

Jan 12 2011 15:24:12 fw-s-svn : %ASA-4-106100: access-list INSIDEACL permitted tcp inside/20.0.0.101(1023) -> outside/172.16.0.251(513) hit-cnt 1 300-second interval [0x2fd140f5, 0x0]

Jan 12 2011 15:24:30 fw-s-svn : %ASA-6-302014: Teardown TCP connection 88236 for outside:172.16.0.251/513 to inside:20.0.0.101/1023 duration 0:00:30 bytes 0 SYN Timeout

Jan 12 2011 15:24:30 fw-s-svn : %ASA-7-609002: Teardown local-host inside:20.0.0.101 duration 0:00:30

an here is a picture of my layout

Kureli Sankar · ‎01-12-2011

Ok. This one syslog says it all

Jan 12 2011 15:24:30 fw-s-svn : %ASA-6-302014: Teardown TCP connection 88236 for outside:172.16.0.251/513 to inside:20.0.0.101/1023 duration 0:00:30 bytes 0 SYN Timeout

The 20.0.0.101 is not responding back or does not have a route to send the SYN ACK back towards the firewall destined to 172.16.0.251

What is this device 20.0.0.101? and it listens on port 513? From a host on that local subnet (172.16.0.x) can you telnet to this IP on port 513? I mean from the run line on another pc or server in the local subnet can you do "telnet 172.16.0.251 513"? Does this work? This 2008 server doesn't have any firewall enabled?

Your other option is to gather captures on the ASA.

cap capin int inside match ip host 20.0.0.101 host 172.16.0.251

cap capout int outside match ip host 20.0.0.101 host 172.16.0.251

sh cap capin det

sh cap capout det

See what packets are seen and whether there is a response coming back from the 2008 server on the outside on the capout.

-KS

MataKiera · ‎01-17-2011

Ok I took that in to advice. I tried to a tel net to the server and the server refused it. OK then I tried to copy my startup-config from the firewall to the server (I have tftp server on the server) and I worked then I tried to copy my startup-config from my router to the server and it failed. Here are the logs.

log

Jan 17 2011 10:15:31 fw-s-svn : %ASA-6-302016: Teardown UDP connection 67 for outside:172.16.0.251/123 to inside:10.0.0.2/123 duration 0:02:01 bytes 48

capin

1: 10:12:26.336454 0024.c417.5a10 0024.14d1.fd82 0x0800 210: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 168 (ttl 255, id 44318)

2: 10:12:26.336484 0024.c417.5a10 0024.14d1.fd82 0x0800 210: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 168 (ttl 255, id 44319)

3: 10:13:30.104959 0024.c417.5a10 0024.14d1.fd82 0x0800 90: 10.0.0.2.123 > 172.16.0.251.123: [udp sum ok] udp 48 [tos 0xc0] (ttl 255, id 0)

4: 10:13:33.336881 0024.c417.5a10 0024.14d1.fd82 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 120 (ttl 255, id 44320)

5: 10:13:33.336927 0024.c417.5a10 0024.14d1.fd82 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 120 (ttl 255, id 44321)

6: 10:14:25.337232 0024.c417.5a10 0024.14d1.fd82 0x0800 114: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 72 (ttl 255, id 44322)

7: 10:14:25.337262 0024.c417.5a10 0024.14d1.fd82 0x0800 114: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 72 (ttl 255, id 44323)

8: 10:14:29.601043 0024.c417.5a10 0024.14d1.fd82 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 0)

9: 10:14:32.601287 0024.c417.5a10 0024.14d1.fd82 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 1)

10: 10:14:36.601287 0024.c417.5a10 0024.14d1.fd82 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 2)

11: 10:14:37.337308 0024.c417.5a10 0024.14d1.fd82 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 120 (ttl 255, id 44324)

12: 10:14:37.337354 0024.c417.5a10 0024.14d1.fd82 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 120 (ttl 255, id 44325)

13: 10:14:41.601302 0024.c417.5a10 0024.14d1.fd82 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 3)

14: 10:14:47.601424 0024.c417.5a10 0024.14d1.fd82 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 4)

capout

1: 10:13:30.105158 0024.14d1.fd85 000c.29b9.a638 0x0800 90: 10.0.0.2.123 > 172.16.0.251.123: [udp sum ok] udp 48 [tos 0xc0] (ttl 255, id 0)

2: 10:13:33.336911 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 120 (ttl 255, id 44320)

3: 10:13:33.336927 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 120 (ttl 255, id 44321)

4: 10:14:25.337247 0024.14d1.fd85 000c.29b9.a638 0x0800 114: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 72 (ttl 255, id 44322)

5: 10:14:25.337278 0024.14d1.fd85 000c.29b9.a638 0x0800 114: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 72 (ttl 255, id 44323)

6: 10:14:29.601302 0024.14d1.fd85 000c.29b9.a638 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 0)

7: 10:14:32.601363 0024.14d1.fd85 000c.29b9.a638 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 1)

8: 10:14:36.601348 0024.14d1.fd85 000c.29b9.a638 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 2)

9: 10:14:37.337323 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 120 (ttl 255, id 44324)

10: 10:14:37.337354 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 120 (ttl 255, id 44325)

11: 10:14:41.601378 0024.14d1.fd85 000c.29b9.a638 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 3)

12: 10:14:47.601500 0024.14d1.fd85 000c.29b9.a638 0x0800 57: 10.0.0.2.53330 > 172.16.0.251.69: [udp sum ok] udp 15 (ttl 255, id 4)

13: 10:15:41.337720 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.3000: [udp sum ok] udp 120 (ttl 255, id 44326)

14: 10:15:41.337751 0024.14d1.fd85 000c.29b9.a638 0x0800 162: 10.0.0.2.56077 > 172.16.0.251.2055: [udp sum ok] udp 120 (ttl 255, id 44327)

PS. Sorry for not answering sooner but I was out of office for a few days

Kureli Sankar · ‎01-18-2011

Unfortunately the discussion ended on Jan 14th. Pls. open a thread on our forum: https://supportforums.cisco.com/community/netpro/security/firewall

to continue this further. Based on the capture, it is clear that the flow is unidirectional and the host 172.16.0.251 doesn't seem to respond back.

-Kureli

jansousedek · ‎01-11-2011

Dear Kureli,

I would like to discuss with you our VPN scenario. In would like to implement VPN remote access (IPSec) using ASA 5510 (ASA 8.3(1), ASDM 6.3(1)). Every client has his own smart card with certificates and private keys on it. I would like to verify the users against Active Directory (AD) database where those certificates are registered. We would like to do it in this way:

1. Computer certificate (non-exportable) will be issued for each laptop, to authenticate computer (so that only certain laptops are allowed) in the first stage of authentication (ASA does this authentication by validating the client computer certificate).

2. In the next step of user authentication, I would like users to use their smart card certificate and enter PIN code to be authenticated against AD.

I am however unsure, whether this scenario is even possible. I am using Cisco VPN Client (5.0.05.0290) to connect to the VPN. I had to deploy MS RADIUS server (on the same domain controller where AD runs) that uses AD database to be able to authenticate users. I was able to install the computer certificate and validate the connection in ASA in the first stage of authentication. However I am only able to authenticate users afterwards using their username and password. On the RADIUS server I have allowed these types of authentication methods: Smart card or other certificate, PEAP, EAP-MSCHAP v2, MS-CHAP v2. ASA is unable to use any of EAP modes and uses only MS-CHAP v2. Is it possible to use EAP? I have entered these commands, but EAP is still not used:

tunnel-group tunnel1 type remote-access
tunnel-group tunnel1 general-attributes
address-pool vpn_pool
authentication-server-group company LOCAL
default-group-policy tunnel1
password-management
tunnel-group tunnel1 ipsec-attributes
trust-point ASDM_TrustPoint4
tunnel-group tunnel1 ppp-attributes
authentication eap-proxy

Is it possible to authenticate users directly against AD (not with NTLM or via RADIUS)?

I was able to use smart card certificate in the first stage of authentication (to verify against ASA certificate), but still I had to type username and password to be able to authenticate against RADIUS server. I was not able to use both certificates (computer certificate and smart card user certificate afterwards). It is possible to do so?

Thank you very much!

harinirina · ‎01-11-2011

Hi,

we've configured our ASA for redundancy (active/standby). we use sub-interface and configured it to be monitored.

The state of the sub-interface is unknown

here is the output of "show failover"

        This host: Primary - Active
                Active time: 7177 (sec)
                  Interface inside (x.x.x.x): Normal
                  Interface outside (x.x.x.x): Normal
                  Interface dmz (x.x.x.x): Normal
                  Interface user (x.x.x.x): Unknown (Waiting)

how can we put this interface to be monitored and have normal state?

Kureli Sankar · ‎01-11-2011

Harini,

Unknown (waiting) is not a good state. This means that it hasn't heard from the other side. Now, what does the other side show?

Do you have standby IP address configured for this sub-interface? Can you from the active unit ping the standby unit and vice versa?

Are you monitoring this interface with the "monitor-interface" command?

How about sh arp for these IP addresses? One unit should have the other ones's mac address it is arp table.

How about the switches arp table? Does that have both units mac address in its arp table as well as mac-addres-table on that vlan?

on the switch:

sh mac-addres-table address vlan

-KS

Kureli Sankar · ‎01-12-2011

I will confirm whether what you are trying to do is possible with double authentication or not.

Regarding LDAP with AD server instead of NTLM or RADIUS - this is doable.

You can follow this link for webvpn which should be very similar for remote access vpn as well.

http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a00808c3c45.shtml

-Kureli

Kureli Sankar · ‎01-14-2011

Hello,

You can only use one cert to complete phase 1. What I have seen done is that windows locks this cert with the smart card and once they use the smart card token to release the cert, then phase one will complete.

It is possible to use LDAP directly against AD.

Trying to use computer cert and then smart card user cert is not possible to do.

This event will be closed today EOB and will be locked. You are welcome to continue this question on our VPN community.

-Kureli

srkrehlik · ‎01-11-2011

Hello Kureli,

you've done a wonderful job answering questions in this thread, so I'll pose an additional one.

I came across a problem similar to the one you had discussed early on regarding a VPN tunnel not passing traffic between failover pairs until the units were rebooted (or failed over) at the head end. This question differs a bit as it was a single PIX525 (ver. 6.3(5)) beign used with Cisco's VPN Client (ver. 4.9 for OS X) initially.

The OS X machine I had conected via the client went to sleep and closed all network connections. I believe this was the cause of the problem as from that point forward I was unable to pass traffic through an established tunnel (P1 / P2 complete, address assigned, routes and DNS propagated to client).

I used an alternate connection method to troubleshoot this while still remote and saw no reason why traffic would not be passing correctly (did not have time to debug fully). Show crypto isakmp sa showed an established connection, show crypto ipsec sa showed a vaild peer with a valid sa and correct ip addressing both locally and remote. Much like in the prior post, I also cleared the associations manually once and made multiple attempts from multiple systems with different clients. In all cases, terminating the connection from the remote client resulted in correct behavior on the PIX, with it removing sa's and incrementing connection id's. However, I was never able to pass traffic from the remote host onto the LAN behind the PIX. Incidentally, a concurrent L2L-ipsec tunnel between that same PIX and one in another office had no problems and was routing traffic correctly.

After a reload command was issued and completed, the unit returned to normal operation.

Is this something you've seen before or possibly even a caveat?

Kureli Sankar · ‎01-11-2011

Sascha,

Thank you for such kind words.

Honestly I have not heard of this. There are some interesting defects that I pointed out on the similar thread that Raman posted that you are referring to.

CSCtd36473 IPsec: Outbound context may be deleted prematurely

CSCtb53186 Duplicate ASP crypto table entry causes firewall to not encrypt traffic

This one I have not heard of. It sure sounds like a defect. I would have to check with a few VPN specialists and get back to you.

I hope you know PIX has been EOL-ed.

Here is the PIX EOL-EOS link:
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/prod_eol_notices_list.html

Here is the 6.3.x code.
http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5708/ps5709/ps2030/end_of_life_notice_for_cisco_pix_sec_app_v63.html

-Kureli

Kureli Sankar · ‎01-11-2011

Sascha,

I have now polled about 3 engineers into this. None of them have seen this issue

Did you gather any data during the time of the problem?

1. sh tech

2. syslogs (debug level)

3. debug cry isa 128

4. debug cry ips 128

5. 'show crypto isakmp sa' and 'show crypto ipsec sa'

It is hard to say now and that too with an older code. It would be better if you open a case with the VPN team should this happen again.

-Kureli