08-05-2013 12:17 AM - edited 03-07-2019 02:44 PM
We currently have 2 nexus 5548 setup for resilience. When OAT testing we discovered that when powering off one of the N5K's and powering on again to test network connectivity. That we got a 30-40 second outage with the current powered on switch when the second one came back on line and the same happened for the othe N5K when we powered that one on and then off.
Has anyone come accross this problem and if so how did you fix it.
Regards
Lee
08-05-2013 12:41 AM
Hi
Can you post your config, if the L2/3 is configured correctly you shouldn't see any dropped packets.
Sent from Cisco Technical Support iPhone App
08-05-2013 01:22 AM
Cheers
version 6.0(2)N1(2)
hostname xxxxxxxxxxxxxx
feature telnet
cfs eth distribute
feature interface-vlan
feature hsrp
feature lacp
feature vpc
feature lldp
feature vtp
clock protocol none
ip domain-lookup
class-map type qos class-fcoe
class-map type queuing class-fcoe
match qos-group 1
class-map type queuing class-all-flood
match qos-group 2
class-map type queuing class-ip-multicast
match qos-group 2
class-map type network-qos class-fcoe
match qos-group 1
class-map type network-qos class-all-flood
match qos-group 2
class-map type network-qos class-ip-multicast
match qos-group 2
vrf context management
ip route 0.0.0.0/0 x.x.40.254
spanning-tree vlan 296-399 priority 24576
vpc domain 1
peer-keepalive destination x.x.42.2 source x.x.42.1 vrf default
delay restore 150
port-profile default max-ports 512
port-profile type port-channel VMware
switchport mode trunk
interface Vlan1
interface Vlan88
no shutdown
ip address x.x.42.33/27
interface Vlan296
no shutdown
ip address x.x.12.250/24
hsrp version 2
hsrp 296
preempt delay minimum 240
priority 150
timers 1 3
ip x.x.12.252
interface Vlan297
no shutdown
ip address x.x.8.250/24
hsrp version 2
hsrp 297
preempt delay minimum 240
priority 150
timers 1 3
ip x.x.8.252
interface Vlan298
no shutdown
ip address x.x.4.250/24
hsrp version 2
hsrp 298
preempt delay minimum 240
priority 150
timers 1 3
ip x.x.4.252
interface Vlan299
no shutdown
ip address x.x.0.250/24
hsrp version 2
hsrp 299
preempt delay minimum 240
priority 150
timers 1 3
ip x.x.0.252
interface Vlan389
no shutdown
ip address x.x.20.250/24
hsrp version 2
hsrp 389
preempt delay minimum 240
priority 150
timers 1 3
ip x.x.20.254
interface port-channel1
switchport mode trunk
switchport trunk allowed vlan 1-380,382-386,389-4094
speed 10000
interface port-channel2
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 1,381,387-388
speed 10000
vpc 2
interface port-channel3
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 1,381,387-388
vpc 3
interface port-channel4
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
vpc 4
interface port-channel5
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
vpc 5
interface port-channel6
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
vpc 6
interface port-channel7
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
vpc 7
interface port-channel8
description xxxxxxxxxxxxxxx
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
vpc 8
interface port-channel88
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
spanning-tree port type network
vpc peer-link
interface Ethernet1/1
interface Ethernet1/2
interface Ethernet1/3
description xxxxxxxxxxxxxxxP1
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 4
interface Ethernet1/4
description xxxxxxxxxxxxxxx-P2
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 4
interface Ethernet1/5
description xxxxxxxxxxxxxxx-P1
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 5
interface Ethernet1/6
description xxxxxxxxxxxxxxx-P2
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 5
interface Ethernet1/7
description xxxxxxxxxxxxxxx-P1
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 6
interface Ethernet1/8
description xxxxxxxxxxxxxxx-P2
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 6
interface Ethernet1/9
description xxxxxxxxxxxxxxx-P1
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 7
interface Ethernet1/10
description xxxxxxxxxxxxxxxP2
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 7
interface Ethernet1/11
description xxxxxxxxxxxxxxx-P1
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 8
interface Ethernet1/12
description xxxxxxxxxxxxxxx-P2
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 8
interface Ethernet1/13
interface Ethernet1/14
interface Ethernet1/15
interface Ethernet1/16
interface Ethernet1/17
switchport mode trunk
switchport trunk allowed vlan 1,381,387-388
channel-group 2 mode active
interface Ethernet1/18
switchport mode trunk
switchport trunk allowed vlan 1,381,387-388
channel-group 3 mode active
interface Ethernet1/19
interface Ethernet1/20
interface Ethernet1/21
interface Ethernet1/22
speed 1000
interface Ethernet1/23
switchport mode trunk
switchport trunk allowed vlan 1-380,382-386,389-4094
channel-group 1 mode active
interface Ethernet1/24
switchport mode trunk
switchport trunk allowed vlan 296-300,381,383,385-389
channel-group 88 mode active
interface Ethernet1/25
interface Ethernet1/26
interface Ethernet1/27
description xxxxxxxxxxxxxxx5
switchport access vlan 300
speed 1000
interface Ethernet1/28
description xxxxxxxxxxxxxxx-0/1
switchport access vlan 299
speed 1000
interface Ethernet1/29
description xxxxxxxxxxxxxxx-0/2
switchport access vlan 299
speed 1000
interface Ethernet1/30
interface Ethernet1/31
description Gamma G1/16
switchport mode trunk
switchport trunk allowed vlan 296-399
speed 1000
interface Ethernet1/32
interface mgmt0
ip address x.x.40.2/24
interface loopback10
ip address x.x.42.1/32
clock timezone gmt -23 0
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.6.0.2.N1.2.bin
boot system bootflash:/n5000-uk9.6.0.2.N1.2.bin
ip route 0.0.0.0/0 x.x.4.254
ip route x.x.0.0/24 x.x.4.254
ip route x.x.42.2/32 x.x.42.34 250
ip route 10.32.0.0/24 x.x.0.248
ip route 10.32.4.0/24 x.x.0.248
ip route 10.32.20.0/24 x.x.0.248
ip route 172.20.56.59/32 x.x.4.254
08-05-2013 11:17 PM
Ok, I can see one problem, you do not have the peer keep alive link in a separate vrf. Your peer keep alive link is currently going across the peer link which is an unsupported design. Either change the vrf to Management(if you have a layer 2 switch connecting both mgmt0 ports) or create a new Peer Keep Alive vrf and change the source/destination command under the vpc domain config to match the new vrf. You need to allocate a /30 subnet for the peer keep alive message, assign one ip address to each N5k. Two ways to do this, create a no switch port(layer 3) or use and SVI, which ever you use remember to add the interface to the new peer keep alive vrf.
Gavin
08-05-2013 11:22 PM
Sorry, I've just looked again and I can't see any port trunking vlan 88. Do you have separate physical cable for peer keep Alives, does your peer keep alive link even show as being up, this would explain your results.
Sent from Cisco Technical Support iPhone App
08-05-2013 11:57 PM
The Vlan 88 vpc peer link has it's own connection.
S2PNGG15N5KC01# sho int vlan 88
Vlan88 is up, line protocol is up
Hardware is EtherSVI, address is 002a.6a36.bebc
Internet Address is 10.16.42.33/27
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
S2PNGG15N5KC01# sho int e1/24
Ethernet1/24 is up
Dedicated Interface
Belongs to Po88
Hardware: 1000/10000 Ethernet, address: 002a.6a36.be7f (bia 002a.6a36.be7f)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 1week(s) 6day(s)
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 80496 bits/sec, 34 packets/sec
30 seconds output rate 12944 bits/sec, 13 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 86.68 Kbps, 26 pps; output rate 12.53 Kbps, 13 pps
RX
55755713 unicast packets 15874261 multicast packets 1334099 broadcast packets
72964073 input packets 17942468270 bytes
3928649 jumbo packets 0 storm suppression bytes
0 runts 0 giants 33 CRC 0 no buffer
33 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
4637474 unicast packets 15475796 multicast packets 11837609 broadcast packets
31950879 output packets 3865575296 bytes
372928 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause S2PNGG15N5KC01# sho int e1/24
Ethernet1/24 is up
Dedicated Interface
Belongs to Po88
Hardware: 1000/10000 Ethernet, address: 002a.6a36.be7f (bia 002a.6a36.be7f)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 1week(s) 6day(s)
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 80496 bits/sec, 34 packets/sec
30 seconds output rate 12944 bits/sec, 13 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 86.68 Kbps, 26 pps; output rate 12.53 Kbps, 13 pps
RX
55755713 unicast packets 15874261 multicast packets 1334099 broadcast packets
72964073 input packets 17942468270 bytes
3928649 jumbo packets 0 storm suppression bytes
0 runts 0 giants 33 CRC 0 no buffer
33 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
4637474 unicast packets 15475796 multicast packets 11837609 broadcast packets
31950879 output packets 3865575296 bytes
372928 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
Regards
Lee
08-06-2013 12:26 AM
vPC keep-alive status : peer is alive
--Peer is alive for : (1190547) seconds, (89) msec
--Send status : Success
--Last send at : 2013.08.06 08:17:26 70 ms
--Sent on interface : Vlan88
--Receive status : Success
--Last receive at : 2013.08.06 08:17:26 52 ms
--Received on interface : Lo10
--Last update from peer : (0) seconds, (458) msec
vPC Keep-alive parameters
--Destination : 10.16.42.2
--Keepalive interval : 1000 msec
--Keepalive timeout : 5 seconds
--Keepalive hold timeout : 3 seconds
--Keepalive vrf : default
--Keepalive udp port : 3200
vpc domain 1
peer-keepalive destination 10.16.42.2 source 10.16.42.1 vrf default
delay restore 150
I believe the peer-keepalive is done from loopback over po1
interface loopback10
ip address 10.16.42.1/32
Regards
08-06-2013 12:21 AM
Yes, po 88 is your Peer link, you need a peer keep alive link. A separate 1gb (can be 10gb also) for the peer keep alive link. Send me the show vpc peer keep alive command output?
Sent from Cisco Technical Support iPhone App
08-06-2013 12:44 AM
Gi Gavin
Could we do the following
Setup another vrf for vpc keepalive and run it over PO88 E1/24 link?
Regards
08-06-2013 12:39 AM
Ok, I've never seen it done like that before. Try moving it to a dedicated vrf. The peer keep alive should not be in the default vrf. Try creating a new vrf and move the loop back up addressing to the new svi(vlan interface) on each n5k, then allow this vlan across a dedicated 1gb or 10 gb trunk link.
Sent from Cisco Technical Support iPhone App
08-06-2013 01:11 AM
Cheers Gavin.
We are going to put dedicated link in for peer-keepalive and setup new vrf for this. Thursday we are doing another oat test so will see if this solve the 30 second outage.
Thanks.
Lee
08-06-2013 01:05 AM
Hi
It doesn't have to be a port channel, just ensure that no other traffic is using the link and it is dedicated to peer keep alive.
Sent from Cisco Technical Support iPhone App
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide