Strange packet losses when traffic is routed via gateway

[Bjorn-Tore Boberg]

Setup is Catalyst C4500X x.x.20.1 (IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.07.00.E), as default gateway for subnet x.x.20.0/24, with static routing to several other gateways.

Testing from host x.x.20.200:

Ping to hosts on same subnet is ok

All ping-test to other subnets is experiencing packet loss, this traffic is to hosts placed behind several different gateways, for instance:
x.x.2.100 via x.x.20.4

x.x.3.20 via x.x.20.3

x.x.40.5 via x.x.20.5

If I configure a route on the host I am testing from (x.x.20.200) to route directly to the destination gateway (instead of the default x.x.20.1), the packet losses disappears:

"route add x.x.3.20 mask 255.255.255.255 x.x.20.3"

So I am pretty certain the problem lies on the default gateway x.x.20.1, but am unable to find anything wrong on this one.

Any idea where to go from here?

[paul driver]

Hello

Can you posrt the configuration for the 4500

res

Paul

[Bjorn-Tore Boberg]

Hello Paul.

Config is now attached.

Regards,

Bjorn-Tore

[paul driver]

Hello

This connectivity testing - Is this on this vrf and  just the one SVI for specifed vlan 200  in your config,

Is this switch suppose to be performing intervlan routing for other vlans?

res

Paul

[Bjorn-Tore Boberg]

Hello

Yes, the testing is on the "myvrf"-vrf with the x.x.20.1 vlan200-SVI .

No intervlan routing is performed on the switch. (not yet at least, we need to sort out these packet losses first)

Regards,

Bjorn-Tore

[devils_advocate]

What are the devices below and what port (s) do they connect to on the 4500? i.e what port does the 4500 forward traffic out of to reach the next hop addresses below?

x.x.20.2
x.x.20.4
x.x.20.3
x.x.20.3
x.x.20.5

[Bjorn-Tore Boberg]

All the next-hop devices are on the vlan200 (x.x.20.0/24), no connectivity issues experienced directly to them from either the C4500X, or from the server used for testing (x.x.20.200). So layer2-connectivity to the next-hop devices seems fine.

x.x.20.2 is a Cisco ASA5525X

x.x.20.3 is another Cisco ASA5525X

x.x.20.4 is a Cisco Catalyst C3560X

x.x.20.5 is a Cisco ASA5510

The packet losses are seen to devices behind all of these, so basically there is packet losses to everything outside the local subnet x.x.20.0/24. But only when the packets are routed via the x.x.20.1 gateway ip on the C4500X.

[Jon Marshall]

You have asymmetric routing here although that isn't necessarily be the issue.

What happens is that your client sends the packet to the 4500 ie. it's default gateway which then has to route the traffic back out of the same interface to one of the other gateways.

However return traffic from the other gateway goes direct to the client because the gateway is in the same IP subnet.

So you may also be seeing ICMP redirects from the 4500 to your clients.

What I would try is to use a different vlan/IP subnet for the client you are pinging from so that all traffic has to go via the 4500 switch.

However this would mean you would needed to update the routing ie. the other gateways would now need a route for the new client subnet pointing to the x.x.20.1 IP on the 4500.

Jon

[Bjorn-Tore Boberg]

That is correct, the C4500X is basically working as a "router-on-a-stick" for the x.x.20.0/24 subnet. In fact, before the C4500X was set up, there was a genuine "router-on-a-stick" doing the same job (a Cisco 2800 router with only one ip address).

C4500X is sending ICMP redirects to the clients as you suspected.

But to my knowledge this should not be a problem, the traffic flow is normal in a router-on-a-stick setup.

To verify completely that the C4500X is the culprit I will try to install the old 2800-router again to verify that the packet losses disappears.