3750G PoE adv-IPsvc IPv6 packet loss on L3 vlan interface

[LELAND VANDERVORT]

Hi All,

We're experiencing a strange situation on a 3750G PoE switch running advanced IP Services featureset whereby frames switched solely at layer 2 (e.g. between IPv6 hosts within the same vlan), but as soon as the frame is forwarded at layer3 through the routed vlan interface, it drops 5% of packets at VERY regular intervals (not random, almost as if it's a timing or buffering issue).  Doing an extended ipv6 ping from the switch itself to a directly attached host we can see this effect clearly as in the following output:

This host is directly connected to the switch and the switch is the default gateway for the vlan itself.

wg-sw1-dc1#ping ipv6
Target IPv6 address: 26XX:XXXX:XXXX:XXXX::XXXX
Repeat count [5]: 500
Datagram size [100]:
Timeout in seconds [2]:
Extended commands? [no]:
Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 26XX:XXXX:XXXX:XXXX::XXXX, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!
!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!
!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!
!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!
!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.
!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!
!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!
!!!!!!!!!!
Success rate is 95 percent (478/500), round-trip min/avg/max = 0/1/26 ms

However, between the switch and an uplink routed interface (physical, not logical vlan interface), there is no packet loss at all.

wg-sw1-dc1#ping ipv6
Target IPv6 address: 26XX:XXXX:YY::ZZ
Repeat count [5]: 500
Datagram size [100]:
Timeout in seconds [2]:
Extended commands? [no]:
Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 26XX:XXXX:YY::ZZ, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 0/2/33 ms

The SDM profile is set to desktop dual ipv4-ipv6 routing.  IOS version is 12.2(55)SE advanced IP Services.

The problem appears to only occur when fowarding to/from one of the VLAN virtual routed interfaces... additionally, this is only occuring on the 3750G PoE -- there is no problem on any of our other production platforms (ranging from 3560, to 3750G [non-PoE], to 6500 with VSS1440).

Anyone happen to have come across anything similar and if so, what work arounds did you find, if any?

Thanks in advance,

Leland

[Fabienne Stephanoff]

Hello Leland,

This almost look like a rate limiter is applied. Is this only occurring when you are pinging an address belonging to the switch? How about two ipv6 hosts on different subnets connected via this switch?

Fabienne

[LELAND VANDERVORT]

Hi Fabienne,

The issue is present both from a host on a VLAN to the switch's vlan interface, as well as any traffic flowing through the layer3 switching process of the switch, so also present across different subnets, or even between a local host and another host elsewhere on the larger network.  Basically anything routed by the switch *and* going across a VLAN virtual layer-3 interface has this problem.

Hosts on the same VLAN/subnet , even though on different physical switch ports, do not exhibit the problem, which is the primary reason that I am chalking this up to something specifically related to IPv6 Layer-3 processing on the switch's VLAN interfaces.   A physical layer-3 switched [routed] port does not exhibit the problem itself. (no problems with IPv4 at all).

Leland