3560 layer 3 failure

[paolo bevilacqua]

Have anybody heard about 3560 switches not processing certain packets at layer 3?

From a 2960 I cannot reach a certain address ONLY. It  goes to the 3560 wireless 1410 bridges that never had a problem.

This happened just as I enabled ip routing after 44 weeks uptime at layer 2.

See below, from the 2960, how the upstream 3560 doesn't handle packet for .54 but it does for .44

There are, of course, no access lists, no special settings of any kind, etc.

AAI-SW-NH>traceroute 192.168.1.44

Type escape sequence to abort.
Tracing the route to 192.168.1.44

  1 192.168.0.5 0 msec 8 msec 0 msec
  2 192.168.3.1 9 msec 0 msec 8 msec
  3 192.168.1.44 17 msec 8 msec 9 msec
AAI-SW-NH>traceroute 192.168.1.54

Type escape sequence to abort.
Tracing the route to 192.168.1.54

  1  *  *  *
  2  *  *  *
  3  *  *  *

And what I think is relevant CEF info taken on the 3560

CommRoomSwitch#sh ip cef 192.168.1.44 internal
192.168.1.0/24, epoch 1, RIB[I], refcount 6, per-destination sharing
  sources: RIB
  feature space:
   Broker: linked
  ifnums:
   Vlan3(967): 192.168.3.1
  path 030B3C2C, path list 030A7C78, share 1/1, type attached nexthop, for IPv4
  nexthop 192.168.3.1 Vlan3, adjacency IP adj out of Vlan3, addr 192.168.3.1 03661740
  output chain: IP adj out of Vlan3, addr 192.168.3.1 03661740
CommRoomSwitch#sh ip cef 192.168.1.54 internal
192.168.1.0/24, epoch 1, RIB[I], refcount 6, per-destination sharing
  sources: RIB
  feature space:
   Broker: linked
  ifnums:
   Vlan3(967): 192.168.3.1
  path 030B3C2C, path list 030A7C78, share 1/1, type attached nexthop, for IPv4
  nexthop 192.168.3.1 Vlan3, adjacency IP adj out of Vlan3, addr 192.168.3.1 03661740
  output chain: IP adj out of Vlan3, addr 192.168.3.1 03661740

Once I change the default gateway on 2960, to use a 1801 router, I can reach the address without any problem.

[Peter Paluch]

Hello Paolo,

Never heard of such behavior before. A couple of ideas:

Is it possible that the ARP cache on the 2960 contains an incorrect entry for the 192.168.1.54, perhaps caused by some ProxyARP in action?

Is it possible that the 2960 uses a different source IP address when it talks to 192.168.1.44 and when it talks to 192.168.1.54?

Can you perform the debug ip icmp and/or the debug ip packet on both the 2960 and the 3560 to see the traceroute probe packets and the ICMP replies? I guess it is worth seeing whether the 3560 receives equivalent packets from the 2960 when performing the traceroute, and whether it reacts to them in a similar way. Of course, use an appropriate ACL to narrow down the packets for which the traceroute is performed.

Best regards,

Peter

[paolo bevilacqua]

Hi Peter, thanks for the suggestions

The 2960 has no arp entry for the problem address, and is not directly connected to it.I did clear arp on all devices and it doesn't appear to be an arp problem.

The 2960 has a single IP address.

I did all kind of traceroutes and troubleshooting coming to the conclusion that the 3560 discards the packet.

I don't know of any troubleshooting command for hardware switching.

As mentioned, as I change the default gateway on 2960 to use a 1801 instead of 3560, everything works.

I will try an update and reload later today.

[Peter Paluch]

Hi Paolo,

You are welcome, although obviously I did not help.

Regarding the debug command to troubleshoot hardware switching, the only command coming to my mind is the show ip cef exact-route but I don't know if that is going to be helpful. Perhaps invalidating the entire CEFand letting the switch reinitialize it using the commands clear adjacency and clear cef table ipv4 would prove helpful.

Nevertheless, packets with expiring TTL should be punted to the CPU and the debug ip packet and debug ip icmp should display such packets or corresponding ICMP Unreachable messages. Are these messages generated at least for the successful traceroute? Having no packets displayed in these debugs at all makes me wonder whether the packets are traversing the 3560 at all.

Best regards,

Peter

[paolo bevilacqua]

I tried these commands to no avails Everythign looks as it should except it doesn't work.

In case of succesful ping/traceroute, the packets are shown by debugs.

On the 2960, debug ip packets show the packet is being sent. And when using a 1801 as default-gw instead of 3560, is handled normally.

[paolo bevilacqua]

I've figured this out. The 2960 had installed an ICMP redirect pointing to an address not in service.

That was the address of the alternate 1801, so testing with it revealed no problem.

Once cleared the redirect, all is good.

[Peter Paluch · Accepted solution]

Hello Paolo,

Thank you very much indeed for your addition and letting us know the source of your problem.

Best regards,

Peter

[paolo bevilacqua]

Thank you Peter for your suggestions and courtesy1