cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
973
Views
0
Helpful
11
Replies

Nexus 5548 resilience problems

ufioperations
Level 1
Level 1

We currently have 2 nexus 5548 setup for resilience.  When OAT testing we discovered that when powering off one of the N5K's and powering on again to test network connectivity. That we got a 30-40 second outage with the current powered on switch when the second one came back on line and the same happened for the othe N5K when we powered that one on and then off.

Has anyone come accross this problem and if so how did you fix it.

Regards

Lee                  

11 Replies 11

Gavin Lodge
Level 1
Level 1

Hi

Can you post your config, if the L2/3 is configured correctly you shouldn't see any dropped packets.

Sent from Cisco Technical Support iPhone App

Cheers

version 6.0(2)N1(2)
hostname xxxxxxxxxxxxxx

feature telnet
cfs eth distribute
feature interface-vlan
feature hsrp
feature lacp
feature vpc
feature lldp
feature vtp
clock protocol none

ip domain-lookup
class-map type qos class-fcoe
class-map type queuing class-fcoe
  match qos-group 1
class-map type queuing class-all-flood
  match qos-group 2
class-map type queuing class-ip-multicast
  match qos-group 2
class-map type network-qos class-fcoe
  match qos-group 1
class-map type network-qos class-all-flood
  match qos-group 2
class-map type network-qos class-ip-multicast
  match qos-group 2

vrf context management
  ip route 0.0.0.0/0 x.x.40.254
spanning-tree vlan 296-399 priority 24576
vpc domain 1
  peer-keepalive destination x.x.42.2 source x.x.42.1 vrf default
  delay restore 150
port-profile default max-ports 512
port-profile type port-channel VMware
  switchport mode trunk


interface Vlan1

interface Vlan88
  no shutdown
  ip address x.x.42.33/27

interface Vlan296
  no shutdown
  ip address x.x.12.250/24
  hsrp version 2
  hsrp 296
    preempt delay minimum 240
    priority 150
    timers  1  3
    ip x.x.12.252

interface Vlan297
  no shutdown
  ip address x.x.8.250/24
  hsrp version 2
  hsrp 297
    preempt delay minimum 240
    priority 150
    timers  1  3
    ip x.x.8.252

interface Vlan298
  no shutdown
  ip address x.x.4.250/24
  hsrp version 2
  hsrp 298
    preempt delay minimum 240
    priority 150
    timers  1  3
    ip x.x.4.252

interface Vlan299
  no shutdown
  ip address x.x.0.250/24
  hsrp version 2
  hsrp 299
    preempt delay minimum 240
    priority 150
    timers  1  3
    ip x.x.0.252

interface Vlan389
  no shutdown
  ip address x.x.20.250/24
  hsrp version 2
  hsrp 389
    preempt delay minimum 240
    priority 150
    timers  1  3
    ip x.x.20.254

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 1-380,382-386,389-4094
  speed 10000

interface port-channel2
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 1,381,387-388
  speed 10000
  vpc 2

interface port-channel3
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 1,381,387-388
  vpc 3

interface port-channel4
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  vpc 4

interface port-channel5
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  vpc 5

interface port-channel6
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  vpc 6

interface port-channel7
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  vpc 7

interface port-channel8
  description xxxxxxxxxxxxxxx

  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  vpc 8

interface port-channel88
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  spanning-tree port type network
  vpc peer-link

interface Ethernet1/1

interface Ethernet1/2

interface Ethernet1/3
  description xxxxxxxxxxxxxxxP1
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 4

interface Ethernet1/4
  description xxxxxxxxxxxxxxx-P2
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 4

interface Ethernet1/5
  description xxxxxxxxxxxxxxx-P1
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 5

interface Ethernet1/6
  description xxxxxxxxxxxxxxx-P2
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 5

interface Ethernet1/7
  description xxxxxxxxxxxxxxx-P1
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 6

interface Ethernet1/8
  description xxxxxxxxxxxxxxx-P2
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 6

interface Ethernet1/9
  description xxxxxxxxxxxxxxx-P1
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 7

interface Ethernet1/10
  description xxxxxxxxxxxxxxxP2
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 7

interface Ethernet1/11
  description xxxxxxxxxxxxxxx-P1
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 8

interface Ethernet1/12
  description xxxxxxxxxxxxxxx-P2
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 8

interface Ethernet1/13

interface Ethernet1/14

interface Ethernet1/15

interface Ethernet1/16

interface Ethernet1/17
  switchport mode trunk
  switchport trunk allowed vlan 1,381,387-388
  channel-group 2 mode active

interface Ethernet1/18
  switchport mode trunk
  switchport trunk allowed vlan 1,381,387-388
  channel-group 3 mode active

interface Ethernet1/19

interface Ethernet1/20

interface Ethernet1/21

interface Ethernet1/22
  speed 1000

interface Ethernet1/23
  switchport mode trunk
  switchport trunk allowed vlan 1-380,382-386,389-4094
  channel-group 1 mode active

interface Ethernet1/24
  switchport mode trunk
  switchport trunk allowed vlan 296-300,381,383,385-389
  channel-group 88 mode active

interface Ethernet1/25

interface Ethernet1/26

interface Ethernet1/27
  description xxxxxxxxxxxxxxx5
  switchport access vlan 300
  speed 1000

interface Ethernet1/28
  description xxxxxxxxxxxxxxx-0/1
  switchport access vlan 299
  speed 1000

interface Ethernet1/29
  description xxxxxxxxxxxxxxx-0/2
  switchport access vlan 299
  speed 1000

interface Ethernet1/30

interface Ethernet1/31
  description Gamma G1/16
  switchport mode trunk
  switchport trunk allowed vlan 296-399
  speed 1000

interface Ethernet1/32

interface mgmt0
  ip address x.x.40.2/24

interface loopback10
  ip address x.x.42.1/32
clock timezone gmt -23 0
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.6.0.2.N1.2.bin
boot system bootflash:/n5000-uk9.6.0.2.N1.2.bin
ip route 0.0.0.0/0 x.x.4.254
ip route x.x.0.0/24 x.x.4.254
ip route x.x.42.2/32 x.x.42.34 250
ip route 10.32.0.0/24 x.x.0.248
ip route 10.32.4.0/24 x.x.0.248
ip route 10.32.20.0/24 x.x.0.248
ip route 172.20.56.59/32 x.x.4.254


Gavin Lodge
Level 1
Level 1

Ok, I can see one problem, you do not have the peer keep alive link in a separate vrf. Your peer keep alive link is currently going across the peer link which is an unsupported design. Either change the vrf to Management(if you have a layer 2 switch connecting both mgmt0 ports) or create a new Peer Keep Alive vrf and change the source/destination command under the vpc domain config to match the new vrf. You need to allocate a /30 subnet for the peer keep alive message, assign one ip address to each N5k. Two ways to do this, create a no switch port(layer 3) or use and SVI, which ever you use remember to add the interface to the new peer keep alive vrf.

Gavin

Gavin Lodge
Level 1
Level 1

Sorry, I've just looked again and I can't see any port trunking vlan 88. Do you have separate physical cable for peer keep Alives, does your peer keep alive link even show as being up, this would explain your results.

Sent from Cisco Technical Support iPhone App

The Vlan 88 vpc peer link has it's own connection.

S2PNGG15N5KC01# sho int vlan 88

Vlan88 is up, line protocol is up

  Hardware is EtherSVI, address is  002a.6a36.bebc

  Internet Address is 10.16.42.33/27

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

S2PNGG15N5KC01# sho int e1/24

Ethernet1/24 is up

Dedicated Interface

  Belongs to Po88

  Hardware: 1000/10000 Ethernet, address: 002a.6a36.be7f (bia 002a.6a36.be7f)

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  Last link flapped 1week(s) 6day(s)

  Last clearing of "show interface" counters never

  1 interface resets

  30 seconds input rate 80496 bits/sec, 34 packets/sec

  30 seconds output rate 12944 bits/sec, 13 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 86.68 Kbps, 26 pps; output rate 12.53 Kbps, 13 pps

  RX

    55755713 unicast packets  15874261 multicast packets  1334099 broadcast packets

    72964073 input packets  17942468270 bytes

    3928649 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  33 CRC  0 no buffer

    33 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    4637474 unicast packets  15475796 multicast packets  11837609 broadcast packets

    31950879 output packets  3865575296 bytes

    372928 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble 0 output discard

    0 Tx pause S2PNGG15N5KC01# sho int e1/24
Ethernet1/24 is up
Dedicated Interface
  Belongs to Po88
  Hardware: 1000/10000 Ethernet, address: 002a.6a36.be7f (bia 002a.6a36.be7f)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 1week(s) 6day(s)
  Last clearing of "show interface" counters never
  1 interface resets
  30 seconds input rate 80496 bits/sec, 34 packets/sec
  30 seconds output rate 12944 bits/sec, 13 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 86.68 Kbps, 26 pps; output rate 12.53 Kbps, 13 pps
  RX
    55755713 unicast packets  15874261 multicast packets  1334099 broadcast packets
    72964073 input packets  17942468270 bytes
    3928649 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  33 CRC  0 no buffer
    33 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    4637474 unicast packets  15475796 multicast packets  11837609 broadcast packets
    31950879 output packets  3865575296 bytes
    372928 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause

Regards

Lee

vPC keep-alive status             : peer is alive                
--Peer is alive for             : (1190547) seconds, (89) msec
--Send status                   : Success
--Last send at                  : 2013.08.06 08:17:26 70 ms
--Sent on interface             : Vlan88
--Receive status                : Success
--Last receive at               : 2013.08.06 08:17:26 52 ms
--Received on interface         : Lo10
--Last update from peer         : (0) seconds, (458) msec

vPC Keep-alive parameters
--Destination                   : 10.16.42.2
--Keepalive interval            : 1000 msec
--Keepalive timeout             : 5 seconds
--Keepalive hold timeout        : 3 seconds
--Keepalive vrf                 : default
--Keepalive udp port            : 3200

vpc domain 1

  peer-keepalive destination 10.16.42.2 source 10.16.42.1 vrf default

  delay restore 150

I believe the peer-keepalive is done from loopback over po1

interface loopback10

  ip address 10.16.42.1/32

Regards

Gavin Lodge
Level 1
Level 1

Yes, po 88 is your Peer link, you need a peer keep alive link. A separate 1gb (can be 10gb also) for the peer keep alive link. Send me the show vpc peer keep alive command output?

Sent from Cisco Technical Support iPhone App

Gi Gavin

Could we do the following

Setup another vrf for vpc keepalive and run it over  PO88 E1/24 link?

Regards

Gavin Lodge
Level 1
Level 1

Ok, I've never seen it done like that before. Try moving it to a dedicated vrf. The peer keep alive should not be in the default vrf. Try creating a new vrf and move the loop back up addressing to the new svi(vlan interface) on each n5k, then allow this vlan across a dedicated 1gb or 10 gb trunk link.

Sent from Cisco Technical Support iPhone App

Cheers Gavin.

We are going to put dedicated link in for peer-keepalive and setup new vrf for this.  Thursday we are doing another oat test so will see if this solve the 30 second outage.

Thanks.

Lee

Gavin Lodge
Level 1
Level 1

Hi

It doesn't have to be a port channel, just ensure that no other traffic is using the link and it is dedicated to peer keep alive.

Sent from Cisco Technical Support iPhone App

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: