4506E with Sup 7e vmotions cause very high latency

[thunt]

After we upgraded our sup iv card in our cat4506e to the new sup 7e, we are having some trouble with very high ping losses throughout various vlans during vmotions.  These are not the usual one or two ping losses associated with the vms in motion.  These are losses to physical as well as virtual machine not involved with the machines in motion.  The latency is so high and so sever that many applications lose connections to other applications and databases. This latency occurs across all vlans.

The ping times intra and inter vlan go from .2 ms to 800 ms or more during vmotions and essentially grinds the network to a halt for several minutes.  This is impacting our production environment (and test and qa).  I have spent several hours on the phone with vmware and they can find nothing wrong with my architecture or my configuration.  This did not happen until we started using the supervisor 7e.  Everything worked flawlessly with the supervisor IV.

Since it is effecting both physical and virtual machines I am suspecting some type of configuration issue with the 4506e.  But what is the most perplexing is that it only happens during vmotions.

I would appreciate any thoughts on the matter. I do have a TAC open but it is taking a long to get any progress.

Thanks for reading.

[thunt]

A little more info

The 2 1Gb nics involved with vmotions are in their own vlan. They are not trunked but in same the same vswitch port group.

interface GigabitEthernet2/43
switchport access vlan 200
switchport mode access
spanning-tree portfast

interface GigabitEthernet3/43
switchport access vlan 200
switchport mode access
spanning-tree portfast

The 2 1Gb nics involved with vm traffic are trunked and in the same dvswitch port groups and etherchanneled on the 4506e.

interface GigabitEthernet2/25
switchport mode trunk
switchport nonegotiate
channel-group 5 mode on
spanning-tree portfast trunk
spanning-tree bpduguard enable

interface GigabitEthernet3/25
switchport mode trunk
switchport nonegotiate
channel-group 5 mode on
spanning-tree portfast trunk
spanning-tree bpduguard enable

interface Port-channel5
switchport
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
spanning-tree bpduguard enable

show etherchannel 5 detail

Group state = L2
Ports: 2   Maxports = 8
Port-channels: 1 Max Port-channels = 1
Protocol:    -
Minimum Links: 0
        Ports in the group:
        -------------------
Port: Gi2/25
------------

Port state    = Up Mstr In-Bndl
Channel group = 5           Mode = On              Gcchange = -
Port-channel  = Po5         GC   =   -             Pseudo port-channel = Po5
Port index    = 0           Load = 0x00            Protocol =    -

Age of the port in the current state: 3d:14h:51m:10s

Port: Gi3/25
------------

Port state    = Up Mstr In-Bndl
Channel group = 5           Mode = On              Gcchange = -
Port-channel  = Po5         GC   =   -             Pseudo port-channel = Po5
Port index    = 1           Load = 0x00            Protocol =    -
         
Age of the port in the current state: 3d:14h:51m:09s
         
                Port-channels in the group:
                ---------------------------
         
Port-channel: Po5
------------
         
Age of the Port-channel   = 73d:15h:39m:18s
Logical slot/port   = 11/5          Number of ports = 2
GC                  = 0x00000000
Port state          = Port-channel Ag-Inuse
Protocol            =    -
Port security       = Disabled
         
Ports in the Port-channel:
         
Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  0     00     Gi2/25   On                 0
  1     00     Gi3/25   On                 0
         
Time since last port bundled:    3d:14h:51m:16s    Gi3/25
Time since last port Un-bundled: 3d:14h:51m:20s    Gi3/25

[thunt]

It also looks like the individual ports in the etherchannel are having(causing??) an issue during the high latency periods.

Feb 18 19:36:52 192.168.x.x 190248: 190302: Feb 19 00:36:51.027: %EC-5-UNBUNDLE: Interface GigabitEthernet2/25 left the port-channel Port-channel5
Feb 18 19:36:52 192.168.x.x 190249: 190303: Feb 19 00:36:51.064: %EC-5-UNBUNDLE: Interface GigabitEthernet3/25 left the port-channel Port-channel5
Feb 18 19:36:55 192.168.x.x 190250: 190304: Feb 19 00:36:54.319: %EC-5-BUNDLE: Interface GigabitEthernet2/25 joined port-channel Port-channel5
Feb 18 19:37:19 192.168.x.x 190251: 190305: Feb 19 00:37:18.633: %EC-5-UNBUNDLE: Interface GigabitEthernet2/25 left the port-channel Port-channel5
Feb 18 19:37:23 192.168.x.x 190252: 190306: Feb 19 00:37:22.319: %EC-5-BUNDLE: Interface GigabitEthernet2/25 joined port-channel Port-channel5
Feb 18 19:37:23 192.168.x.x 190253: 190307: Feb 19 00:37:22.335: %EC-5-BUNDLE: Interface GigabitEthernet3/25 joined port-channel Port-channel5
Feb 18 19:37:27 192.168.x.x 190254: 190308: Feb 19 00:37:26.439: %EC-5-UNBUNDLE: Interface GigabitEthernet2/25 left the port-channel Port-channel5
Feb 18 19:37:27 192.168.x.x 190255: 190309: Feb 19 00:37:26.483: %EC-5-UNBUNDLE: Interface GigabitEthernet3/25 left the port-channel Port-channel5
Feb 18 19:37:31 192.168.x.x 190256: 190310: Feb 19 00:37:30.319: %EC-5-BUNDLE: Interface GigabitEthernet2/25 joined port-channel Port-channel5
Feb 18 19:37:31 192.168.x.x 190257: 190311: Feb 19 00:37:30.337: %EC-5-BUNDLE: Interface GigabitEthernet3/25 joined port-channel Port-channel5
Feb 18 19:38:57 192.168.x.x 190258: 190312: Feb 19 00:38:56.627: %EC-5-UNBUNDLE: Interface GigabitEthernet2/25 left the port-channel Port-channel5
Feb 18 19:38:57 192.168.x.x 190259: 190313: Feb 19 00:38:56.656: %EC-5-UNBUNDLE: Interface GigabitEthernet3/25 left the port-channel Port-channel5
Feb 18 19:39:00 192.168.x.x 190260: 190314: Feb 19 00:38:59.319: %EC-5-BUNDLE: Interface GigabitEthernet2/25 joined port-channel Port-channel5
Feb 18 19:39:00 192.168.x.x 190261: 190315: Feb 19 00:39:00.319: %EC-5-BUNDLE: Interface GigabitEthernet3/25 joined port-channel Port-channel5