Pix Not responding to Internal ICMP

mumbles202 · ‎08-22-2013

Working on a Pix with a pretty vanilla configuration. It is a 515e with a failover only license and from what I understand if I disable failover it should be configurable and usable during a migration (with the caveat that it will reboot every 24 hours). So I've uploaded the config and verified it is all intact and in place after a power cycle. But if I try to ping the pix from the 3560 that is directly connected to it I get no replies. On the switch I'm seeing incomplete arp entries. And on the firewall I'm seeing the packet count increase as ping it, just no replies. Tried different ports on the switch, different ports on the pix. All yield the same results. I've used the switch a few days ago so I think it is good. Tried different patch cables, tried crossover as well.

Harvey Ortiz · ‎08-23-2013

Hi david,

can you provide the ASA/Switch ports configuration?

Also run:

show run icmp

are those interfaces on the same network?

Regards,

Harvey.

mumbles202 · ‎09-03-2013

I'll pull the "sh run icmp" and post it. I'll get the configuration of the ports as well. The switch and pix are on the same network (created a vlan interface on the switch, plugged in pix to port in that vlan. Can ping interface from laptop plugged into another port on that vlan).

mumbles202 · ‎09-03-2013

Here is the relevant config:

PIX Config relevant to inside

interface ethernet1 auto

nameif ethernet1 inside security100

ip address inside 192.168.100.5 255.255.255.0

no failover ip address inside

Switchport Config

interface Vlan20

ip address 192.168.100.10 255.255.255.0

end

interface FastEthernet0/19

description Connection to Pix

switchport access vlan 20

switchport mode access

end

The "sh run icmp" displayed all the configuration for the pix so I didn't post it. I did try a

"icmp permit 192.168.100.0 255.255.255.0 inside" but that yields the same results. If I do a "sh ip arp" on my switch I get:

Internet 192.168.100.5 0 Incomplete ARPA

If I do a sh interface ethernet1 on the pix i do see the counters incrementing as I ping the interface.

Harvey Ortiz · ‎09-03-2013

hello david,

From the outputs the switch got the ARP entry of the PIX, verify the same on the PIX:

show arp | inc 192.168.100.10

Then we will know if the pix has the ARP entry within the switch mac address.

Just in case you can place some captures on the inside interface of the pix:

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080a9edd6.shtml#cliconfig

access-list TEST extended permit ip host 192.168.100.10 host 192.168.100.5

access-list TEST extended permit ip host 192.168.100.5 host 192.168.100.10

capture capin interface inside match access-list TEST

On this way you would verify if the pix is receiving icmp packets from the switch(192.168.100.10)

If possible you can try to reboot the switch and test again.

Regards,

Harvey

mumbles202 · ‎09-03-2013

Thanks. Forgot to mention that the mac of the switch is in the arp table of the pix. I added the following to the pix:

access-list test permit ip host 192.168.100.10 host 192.168.100.5

access-list test permit ip host 192.168.100.5 host 192.168.100.10

capture capin access-list test interface inside

Then did a ping from my switch (which is directly connected to ethernet1 w/ a straight-through patch cable). After that I did a sh capture capin

PIX515E(config)# sh capture capin

0 packet captured

0 packet shown

I did notice that if I do a "sh run | in capture" I don't return any lines of code w/ the capture statement.

Harvey Ortiz · ‎09-03-2013

We should see the icmp packets arriving to the inside interface of the pix, the capture doesn't show anything.

Let's try:

debug icmp

and let me know.

Disable debugs:

undebug all

mumbles202 · ‎09-03-2013

Only option on the pix is debug icmp trace. If i enable that and try to ping from the switch I see nothing. If I ping from the pix I get this:

7: ICMP echo request (len 32 id 9233 seq 0) 192.168.100.5 > 192.168.100.10

192.168.100.10 NO response received -- 1000ms

8: ICMP echo request (len 32 id 9233 seq 1) 192.168.100.5 > 192.168.100.10

192.168.100.10 NO response received -- 1000ms

9: ICMP echo request (len 32 id 9233 seq 2) 192.168.100.5 > 192.168.100.10

192.168.100.10 NO response received -- 1000ms

Harvey Ortiz · ‎09-05-2013

Well from that output the pix is able to generate icmp packets, but it seems that the switch is not replying back

I would recommend to create on the switch an extended access-list for source/destination you are pinging.

eg;

access-list 101 permit icmp host 192.168.100.5 host 192.168.100.10

debug ip packet 101 detail

The above will only show detailed ICMP debugging between the 2 hosts specified in the ACL.

Also can you attach the show interface

I would recommend to create an extended access-list for source/destination you are pinging.

eg;

access-list 101 permit icmp host 11.11.11.11 host 22.22.22.22

debug ip packet 101 detail

The above will only show detailed ICMP debugging between the 2 hosts specified in the ACL.

Also can you attach the show interface ethernet1 and show interfaces fastethernet0/19

mumbles202 · ‎09-05-2013

Here is the output of the interfaces commands:

FastEthernet0/19 is up, line protocol is up (connected)

Hardware is Fast Ethernet, address is 0018.ba50.ff15 (bia 0018.ba50.ff15)

MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, media type is 10/100BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts (0 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

599 packets output, 47738 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

PIX515E# sh int ethernet1

interface ethernet1 "inside" is up, line protocol is up

Hardware is i82559 ethernet, address is 000f.904b.8208

IP address 192.168.100.5, subnet mask 255.255.255.0

MTU 1500 bytes, BW 100000 Kbit full duplex

53 packets input, 3198 bytes, 0 no buffer

Received 53 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 packets output, 0 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

input queue (curr/max blocks): hardware (128/128) software (0/1)

output queue (curr/max blocks): hardware (0/0) software (0/0)

I connected my laptop to the switch and when I ping the pix i do see the arp entry in the pix, just get no response.

Debug:

*Mar 1 00:37:42.692: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:42.692: ICMP type=8, code=0, RCLI(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:42.692: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:42.692: ICMP type=8, code=0, Local Clustering(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:42.692: IP: tableid=0, s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), routed via RIB

*Mar 1 00:37:42.692: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, sending

*Mar 1 00:37:42.692: ICMP type=8, code=0

*Mar 1 00:37:42.692: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, output feature

*Mar 1 00:37:42.692: ICMP type=8, code=0, Check hwidb(72), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE .

*Mar 1 00:37:45.712: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, encapsulation failed

*Mar 1 00:37:45.712: ICMP type=8, code=0

*Mar 1 00:37:45.712: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:45.712: ICMP type=8, code=0, RCLI(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:45.712: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:45.712: ICMP type=8, code=0, Local Clustering(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:45.712: IP: tableid=0, s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), routed via RIB

*Mar 1 00:37:45.712: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, sending

*Mar 1 00:37:45.712: ICMP type=8, code=0

*Mar 1 00:37:45.712: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, output feature

*Mar 1 00:37:45.712: ICMP type=8, code=0, Check hwidb(72), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE .

*Mar 1 00:37:48.732: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, encapsulation failed

*Mar 1 00:37:48.732: ICMP type=8, code=0

*Mar 1 00:37:48.732: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:48.732: ICMP type=8, code=0, RCLI(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:48.732: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:48.732: ICMP type=8, code=0, Local Clustering(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:48.732: IP: tableid=0, s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), routed via RIB

*Mar 1 00:37:48.732: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, sending

*Mar 1 00:37:48.732: ICMP type=8, code=0

*Mar 1 00:37:48.732: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, output feature

*Mar 1 00:37:48.732: ICMP type=8, code=0, Check hwidb(72), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE .

*Mar 1 00:37:51.752: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, encapsulation failed

*Mar 1 00:37:51.752: ICMP type=8, code=0

*Mar 1 00:37:51.752: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:51.752: ICMP type=8, code=0, RCLI(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:51.752: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:51.752: ICMP type=8, code=0, Local Clustering(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:51.752: IP: tableid=0, s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), routed via RIB

*Mar 1 00:37:51.752: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, sending

*Mar 1 00:37:51.752: ICMP type=8, code=0

*Mar 1 00:37:51.752: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, output feature

*Mar 1 00:37:51.752: ICMP type=8, code=0, Check hwidb(72), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE .

*Mar 1 00:37:54.772: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, encapsulation failed

*Mar 1 00:37:54.772: ICMP type=8, code=0

*Mar 1 00:37:54.772: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:54.772: ICMP type=8, code=0, RCLI(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:54.772: IP: s=192.168.100.10 (local), d=192.168.100.5, len 100, local feature

*Mar 1 00:37:54.772: ICMP type=8, code=0, Local Clustering(8), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

*Mar 1 00:37:54.772: IP: tableid=0, s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), routed via RIB

*Mar 1 00:37:54.772: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, sending

*Mar 1 00:37:54.772: ICMP type=8, code=0

*Mar 1 00:37:54.772: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, output feature

*Mar 1 00:37:54.772: ICMP type=8, code=0, Check hwidb(72), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE .

Success rate is 0 percent (0/5)

Switch#

*Mar 1 00:37:57.792: IP: s=192.168.100.10 (local), d=192.168.100.5 (Vlan1), len 100, encapsulation failed

*Mar 1 00:37:57.792: ICMP type=8, code=0

mumbles202 · ‎09-05-2013

Decided to turn on arp debugging as well on both devices. On the PIX I'm seeing this:

94: arp-in: request at inside from 192.168.100.10 0018.ba50.ff40 for 192.168.100.5 0000.0000.0000

95: arp-set: added arp inside 192.168.100.10 0018.ba50.ff40

96: arp-in: generating reply from 192.168.100.5 000f.904b.8208 to 192.168.100.10 0018.ba50.ff40

97: arp-in: request at inside from 192.168.100.10 0018.ba50.ff40 for 192.168.100.5 0000.0000.0000

98: arp-set: added arp inside 192.168.100.10 0018.ba50.ff40

99: arp-in: generating reply from 192.168.100.5 000f.904b.8208 to 192.168.100.10 0018.ba50.ff40

100: arp-in: request at inside from 192.168.100.10 0018.ba50.ff40 for 192.168.100.5 0000.0000.0000

101: arp-set: added arp inside 192.168.100.10 0018.ba50.ff40

102: arp-in: generating reply from 192.168.100.5 000f.904b.8208 to 192.168.100.10 0018.ba50.ff40

103: arp-in: request at inside from 192.168.100.10 0018.ba50.ff40 for 192.168.100.5 0000.0000.0000

104: arp-set: added arp inside 192.168.100.10 0018.ba50.ff40

105: arp-in: generating reply from 192.168.100.5 000f.904b.8208 to 192.168.100.10 0018.ba50.ff40

On the Switch I'm getting this:

*Mar 1 01:11:37.567: IP ARP: creating incomplete entry for IP address: 192.168.100.5 interface Vlan1

*Mar 1 01:11:37.567: IP ARP: sent req src 192.168.100.10 0018.ba50.ff40,

dst 192.168.100.5 0000.0000.0000 Vlan1

*Mar 1 01:11:38.574: IP ARP throttled out the ARP Request for 192.168.100.5.

Success rate is 0 percent (0/1)

Switch#

*Mar 1 01:11:39.580: IP ARP: sent req src 192.168.100.10 0018.ba50.ff40,

dst 192.168.100.5 0000.0000.0000 Vlan1

*Mar 1 01:11:40.587: IP ARP throttled out the ARP Request for 192.168.100.5

Harvey Ortiz · ‎09-06-2013

Well from the debugs the pix seems to be OK, I think the issue is on the switch (IP ARP: creating incomplete entry for IP address: 192.168.100.5 interface Vlan1)

Take a look on this document explaining the reason of the debug message

https://supportforums.cisco.com/docs/DOC-2094

Let me know if that works.

Regards,

Harvey

mumbles202 · ‎09-11-2013

Thanks for the feedback. I was thinking of the same thing, so I tried another known-good switch that ended up having the same issue. Tried multiple ports on both switches. Tried multiple ports on the pix as well (has a 4 port card in it).

Just to be sure I just tried a 3rd switch which also did the same. Then i used a cross-over cable btwn my laptop and the pix and tested and I saw the arp entry on the pix but no reply.

Did some more checking and was able to get it to respond after I issued the "failover active" command. This has a failover only license but will be used as a cold spare.