Hi,Dear Sir/Madam,
we configured VPC under 7K, and using otv connect to two site
recently we upgrade software and reload one by one for 4 netxus 7K ,and found some problems.
for example
If we reload nexus7k-1, and check log on nexus7k-2. we found connected to nexus7k-1 were down, this is normal, but after several seconds, all interface port channel of PVC on nexus7k-2 are down and after 7-10 seconds all interface port channel up again.
if we login otv VDC or one downside vpc member switch, this VDC is vpc member of two nexus 7k, if we reload nexus7k-1, we can find interface connected to nexus7k-1 was going down , this is normal , but after few seconds, anther interface connected to nexus7k-2 was also going down , the result was interface port channel was going down and resume after 1 minutes. during this time , all traffic was lost.
why we reload nexus7k-1, why can cause downside vpc member switch another interface connected to nexus7k-2 was going down? then whole portchannel is going down , and traffic was totally lost for almost 2 minutes
below is log file of otv-1 VDC
otv-1 VDC is a member of vpc of nexus7k-1 and nexus7k-2,port 3/48 connect to nexus7k-2 , port 2/48 connect to nexus 7k-1 belong to same vpc and port channel 5.
we reloaded nexus7k-2. port 3/48 on otv-1 was going down ,this normal
2017 Jul 13 04:03:49 UTC: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel5: Ethernet3/48 is down
2017-07-13 04:03:49 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:49 UTC: %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel5,bandwidth changed to 10000000 Kbit
2017 Jul 13 04:03:49 UTC: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet3/48 is down (Link failure)
we can see port 3/48 was going down , and portchannel5 bandwidth changed from 20G to half, 10G, this is normal
but after just only 4 seconds
2017 Jul 13 04:03:53 UTC: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel5: first operational port changed from Ethernet2/48 to none
2017-07-13 04:03:53 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:53 UTC: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel5 is down (No operational members)
2017-07-13 04:03:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:54 UTC: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel5: Ethernet2/48 is down
2017-07-13 04:03:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:54 UTC: %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel5,bandwidth changed to 100000 Kbit
2017-07-13 04:03:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:54 UTC: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet2/48 is down (Link failure)
2017-07-13 04:03:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:03:54 UTC: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel5 is down (No operational members)
port 2/48 on otv-1 VDC was going down no reason,reported Ethernet2/48 is down(Link failure) ,why? cause port-channel 5 bandwidth change to 100M ,lost all member, and was also going down, but nexus7k-2 was still running.
So, all traffic through otv-1 VDC was lost, and product network was going down, very serious problem, because before we reload nexus7k-2. we though we have nexus7k-1 and vpc, we have one path to the outside although we reload nexus7k-2
after that
2017 Jul 13 04:04:54 UTC: %ETHPORT-5-SPEED: Interface Ethernet2/48, operational speed changed to 10 Gbps
2017-07-13 04:04:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:04:54 UTC: %ETHPORT-5-IF_DUPLEX: Interface Ethernet2/48, operational duplex mode changed to Full
2017-07-13 04:04:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:04:54 UTC: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet2/48, operational Receive Flow Control state changed to off
2017-07-13 04:04:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:04:54 UTC: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet2/48, operational Transmit Flow Control state changed to off
2017-07-13 04:04:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:04:54 UTC: %ETHPORT-5-SPEED: Interface port-channel5, operational speed changed to 10 Gbps
2017-07-13 04:04:54 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:04:54 UTC: %ETHPORT-5-IF_DUPLEX: Interface port-channel5, operational duplex mode changed to Full
after that the port 2/48 and port-channel 5 was seem re-initial interface and came up again after one minute,
because we reloaded nexus7k-2 ,lost otv-2 VDC and now otv-1 VDC ethernet2/48 and port-channel down for one minute , so all traffic stopped for
almost 2 minutes.
eventually , nexus7k-2 come back after 8 minutes,
2017 Jul 13 04:12:51 UTC: %ETH_PORT_CHANNEL-5-PORT_UP: port-channel5: Ethernet3/48 is up
2017-07-13 04:12:50 Local7.Notice 10.149.0.26 : 2017 Jul 13 04:12:51 UTC: %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel5,bandwidth changed to 20000000 Kbit
this normal.
we test several times, as we reload one nexus 7k, we can see the other interface in same VPC was going down and up ,then we lost connectivity for 2 minutes.
this is weired , and not acceptable.
thank you!
Tom
