cisco WS-C3750X-48 10G module | VMware host intermittent connectivity

[maraica]

Hi everyone, I have a weird issue going on in our data center.  I recently upgraded the network module on a cisco switch from 4|1gb ports to 2|1 GB  2|10Gb.  The 2|1gb are shut.  I did create an etherchannel/portchannel to the aggregation cores switches with 2|10GB ports.  Everything is working like a charm except that a couple of hosts in the vmaware environment do have an intermittent connectivity.  Infrastructure engineers think that it was due to my recent upgrade.  How can I validate that there is nothing wrong with the switch.  They said that the reason may be because I am running LACP on the etherchannel.  We have a few servers/services on this switch and the only thing having issues is these vmware hosts.  

#show int gi6/1/1
GigabitEthernet6/1/1 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 442b.036f.73b1 (bia 442b.036f.73b1)
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Auto-duplex, Auto-speed, link type is auto, media type is Not Present
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

#show int gi6/1/2
GigabitEthernet6/1/2 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 442b.036f.73b2 (bia 442b.036f.73b2)
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Auto-duplex, Auto-speed, link type is auto, media type is Not Present
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer

#show int port-channel6
Port-channel6 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 442b.036f.73b5 (bia 442b.036f.73b5)
Description: ** ETHERCHANNEL TRUNK TO CORES **
MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Te6/1/1 Te6/1/2
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:04:34, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 8332000 bits/sec, 2112 packets/sec
5 minute output rate 12625000 bits/sec, 2375 packets/sec
16932679777 packets input, 6341412374696 bytes, 0 no buffer
Received 56461736 broadcasts (37295298 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 37295298 multicast, 0 pause input
0 input packets with dribble condition detected
44351494500 packets output, 50604476180411 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

#show ether summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
6 Po6(SU) LACP Te6/1/1(P) Te6/1/2(P)

[marce1000]

 - How is the intermittent connectivity experienced  ? Also activate status logging on the ports of the etherchannel with logging event link-status. Check the switch logs on  a regular basis (show logging) and or configure a syslog server to collect logging-data from the switch.

M.

[maraica]

Marce, the switch does not show any log errors.  The ports lights stay solid green (no orange or on/off).

Per Infrastructure Engineers, within vmware, one or two hosts only out 20 or so goes green and yellow (no available)  Then it will become available.  This behavior is not within minutes..the hosts may show down or up within hours.  

[Jaderson Pessoa]

Guy, your configurations its corrects.

Could you post: show run int te 6/1/1 and 2?

On vmware there a feature that you need check it (promiscous mode), learn more about: https://kb.vmware.com/s/article/1002934

[maraica]

#sho int te6/1/1
TenGigabitEthernet6/1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 442b.036f.73b5 (bia 442b.036f.73b5)
Description: ** ETHERCHANNEL 6 TO CORES **
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-CX1
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:10, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 89459000 bits/sec, 8848 packets/sec
5 minute output rate 9156000 bits/sec, 1120 packets/sec
9478848821 packets input, 3547399244624 bytes, 0 no buffer
Received 19715490 broadcasts (12137177 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 12137177 multicast, 0 pause input
0 input packets with dribble condition detected
7177378198 packets output, 8430243680985 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

#sho int te6/1/2
TenGigabitEthernet6/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 442b.036f.73b6 (bia 442b.036f.73b6)
Description: ** ETHERCHANNEL 6 TO CORES **
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-CX1
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:16, output 00:00:06, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1226000 bits/sec, 570 packets/sec
5 minute output rate 1892000 bits/sec, 2079 packets/sec
7603940844 packets input, 2934088133391 bytes, 0 no buffer
Received 37099410 broadcasts (25404656 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 25404656 multicast, 0 pause input
0 input packets with dribble condition detected
37266682553 packets output, 42231336572495 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

[maraica]

!
interface TenGigabitEthernet6/1/1
description ** ETHERCHANNEL 6 TO CORES **
switchport trunk encapsulation dot1q
switchport trunk native vlan xx
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active
!
interface TenGigabitEthernet6/1/2
description ** ETHERCHANNEL 6 TO CORES **
switchport trunk encapsulation dot1q
switchport trunk native vlan xx
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active

[Reza Sharifi]

To validate the Portchannel is not an issue, you can change the mode from LACP to mode On/On.

The other thing you may want to look at is the criteria for load balancing. I think the 3750 is source mac but the VMs are source IP.

HTH

[maraica]

Thanks Reza, do I do this int the portchanel or at the interfaces?  do you know the specific command?

mode from LACP to mode On/On.

[Reza Sharifi]

Hi,

You would have to switch from LACP to On/ON in a maintenance window as it is disruptive.

So, under each interface, you negate the LACP first and than add mod on.

example

interface TenGigabitEthernet6/1/1
description ** ETHERCHANNEL 6 TO CORES **
switchport trunk encapsulation dot1q
switchport trunk native vlan xx
switchport mode trunk
switchport nonegotiate
channel-group 6 mode active

config t

interface TenGigabitEthernet6/1/1

no channel-group 6 mode active

channel-group 6 mode on

Make sure to negate it on both interfaces first if not you will see an error.

HTH

[Reza Sharifi]

One more thing. Make sure to do one change at a time. Do the Portchannel first and then test with the VM guys.

If that does not help, you may want to look at the load balancing criteria next and test.

HTH

[maraica]

Reza,

when you said "Make sure to negate it on both interfaces first if not you will see an error" you imply to do the command on both interfaces on the same switch (6/1/1 and 6/1/2), but no changes are needed at the core/aggregation switch?

thank you

[Reza Sharifi]

Hi,

it needs to be applied on both core/aggregation and as well as the other side.  So, 6/1/1,6/1/2 and whatever interfaces these 2 interfaces connect to. So, negate all 4 interfaces on both sides and then add mode ON/ON on all.

Please remember that the change is disruptive and should be performed in a maintenance window and not during production hours.

HTH

[maraica]

Just want to let everyone know that after removing the LACP etherchannel, the issue with vmware hosts disconnecting continued.  I did rebuild the etherchannels on LACP ( this is what vmware recommends anyway).  

[KLimSLKJJSL2000]

Hello @maraica 

We are experiencing similar problems, did you find a solution in the end and if so what was it ?

Thank you