Solved: HSRP instability

Mauricio Naegele da Cruz · ‎01-26-2011

Hello everyone,

I'm having a problem of hsrp instability. The scenario is two 6500 and one port-channel ( two tengiga ports - 6704). I started to notice hsrp instability and searching for one workaround i´ve found the ip spd command. This command has ended with the flapping, but i´m still looking for the cause of the problem.

Any suggestions ???

best regards,

Mauricio

Satya Narra · ‎01-28-2011

Input queue: 0/75/112/0 (size/max/drops/flushes); Total output drops: 0

Q depth is just 75,

try hold-queue 2000 on the vlan interface. It should resolve it.

But, also go through the following documentation.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic2

In case if it does not resolve the issue, we need to find out what kind of traffic is going to the CPU at the time of drops.

Have a great day.

View solution in original post

Amit Singh · ‎01-26-2011

Could you Please paste the interface configurations and logs if you have any?

Cheers,

-amit singh

rassoul.ghaznavi · ‎01-26-2011

Are you using copper or fiber?

Have you checked the negotiationon the link?

Satya Narra · ‎01-27-2011

Hi,

Since you say that ip spd has solved da issue, it looks like you were having input Q drops on the interface. Can you give us more info like interface config and show interface o/p and da config you put in. I will get back to you with a better answer.

Sent from Cisco Technical Support iPhone App

Satya Narra · ‎01-27-2011

check out the SPD document

http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a008012fb87.shtml

This should explain why SPD change resolved the issue.

Mauricio Naegele da Cruz · ‎01-27-2011

Here is the config and some shows :

Ios Version : 12.2(17d)SXB11a

##global##

ip spd queue max-threshold 600

ip spd queue min-threshold 500

###Switch A###

interface TenGigabitEthernetx/y

mtu 9216

no ip address

mls qos trust dscp

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

!

TenGigabitEthernetx/y is up, line protocol is up (connected)

Hardware is C6k 10000Mb 802.3, address is 0025.842f.322e (bia 0025.842f.322e)

Description:

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 87/255, rxload 125/255

Encapsulation ARPA, loopback not set

Full-duplex, 10Gb/s

input flow-control is off, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:03, output 00:00:26, output hang never

Last clearing of "show interface" counters 5d19h

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 4936424000 bits/sec, 2480996 packets/sec

5 minute output rate 3440547000 bits/sec, 1880687 packets/sec

686292835356 packets input, 183937804559174 bytes, 0 no buffer

Received 179655086 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

520451505701 packets output, 130350475370460 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

!

SWITCH_A#sh ip spd

Current mode: normal.

Queue min/max thresholds: 500/600, Headroom: 100

IP normal queue: 0, priority queue: 1.

SPD special drop mode: none

###Switch B###

interface TenGigabitEthernetZ/K

mtu 9216

no ip address

mls qos trust dscp

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

!

TenGigabitEthernetZ/K is up, line protocol is up (connected)

Hardware is C6k 10000Mb 802.3, address is 0025.45f1.adf2 (bia 0025.45f1.adf2)

Description:

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 125/255, rxload 87/255

Encapsulation ARPA, loopback not set

Full-duplex, 10Gb/s

input flow-control is off, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:12, output 00:00:49, output hang never

Last clearing of "show interface" counters 5d18h

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 3425071000 bits/sec, 1878092 packets/sec

5 minute output rate 4921813000 bits/sec, 2482505 packets/sec

519735420778 packets input, 130210672707506 bytes, 0 no buffer

Received 88475812 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

685338021183 packets output, 183730030043381 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

SWITCH_B#sh ip spd

Current mode: normal.

Queue min/max thresholds: 500/600, Headroom: 100

IP normal queue: 0, priority queue: 0.

SPD special drop mode: none

Satya Narra · ‎01-27-2011

The show interfaces are very clean. Did you capture these output after the issue has been resolved?

I see that interfaces are just configured as trunk. I am expecting that vlan interfaces have the HSRP configuration, is that right?

Can you check the vlan interfaces for any input Q drops?

Mauricio Naegele da Cruz · ‎01-28-2011

This is correct, the output was collected yesterday afternoon. I forgot to mention that when i notice the problem the traffic it was close to 5 giga (peak traffic). Yesterday at night the problem has occurred again even with the spd configuration.

Here are some information about the vlans that i´ve collected rigth now:

###SWITCH_A###

interface Vlan12

ip address x.x.x.2 255.255.255.0

no ip redirects

standby 12 ip x.x.x.1

standby 12 preempt

!

Vlan12 is up, line protocol is up

Hardware is EtherSVI, address is 0015.2cb6.ca00 (bia 0015.2cb6.ca00)

Description: Rede GERENCIA II

Internet address is x.x.x.2

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 25/255, rxload 1/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:01, output hang never

Last clearing of "show interface" counters 6d11h

Input queue: 0/75/112/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 9000 bits/sec, 16 packets/sec

5 minute output rate 99224000 bits/sec, 13247 packets/sec

L2 Switched: ucast: 5136130694 pkt, 6031008158267 bytes - mcast: 10835447 pkt, 1082019516 bytes

L3 in Switched: ucast: 45764 pkt, 2947309 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 9275671189 pkt, 7145521811463 bytes mcast: 0 pkt, 0 bytes

9418387 packets input, 664720408 bytes, 0 no buffer

Received 9355383 broadcasts, 0 runts, 0 giants, 2 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

9294236023 packets output, 7125481295613 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

interface Vlan70

mtu 1526

ip address x.x.x.162 255.255.255.240

no ip redirects

no ip unreachables

ip mtu 1500

load-interval 30

standby 70 ip x.x.x.161

standby 70 priority 105

standby 70 preempt

!

Vlan70 is up, line protocol is up

Hardware is EtherSVI, address is 0015.2cb6.ca00 (bia 0015.2cb6.ca00)

Description: Vlan - GBoIP_BSC_SNE01

Internet address is x.x.x.162

MTU 1526 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 7/255, rxload 1/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:01, output hang never

Last clearing of "show interface" counters 6d11h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 2000 bits/sec, 5 packets/sec

30 second output rate 29775000 bits/sec, 8586 packets/sec

L2 Switched: ucast: 2250492201 pkt, 421466690051 bytes - mcast: 3010411 pkt, 184055808 bytes

L3 in Switched: ucast: 23908 pkt, 4210410 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 4801497663 pkt, 2202834662978 bytes mcast: 0 pkt, 0 bytes

3053897 packets input, 200937411 bytes, 0 no buffer

Received 3010403 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

4805260906 packets output, 2184850651347 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

!

###SWITCH_B###

interface Vlan12

ip address x.x.x.3 255.255.255.0

no ip redirects

standby 12 ip x.x.x.1

standby 12 priority 105

standby 12 preempt

end

SWITCH_B#sh int vlan 12

Vlan12 is up, line protocol is up

Hardware is EtherSVI, address is 0015.2cb6.de00 (bia 0015.2cb6.de00)

Description:

Internet address is x.x.x.3/24

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 2/255, rxload 31/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 6d11h

Input queue: 0/75/95/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 124290000 bits/sec, 15035 packets/sec

5 minute output rate 9339000 bits/sec, 964 packets/sec

L2 Switched: ucast: 16960444670 pkt, 15204885675072 bytes - mcast: 10882456 pkt, 1087842971 bytes

L3 in Switched: ucast: 9958793719 pkt, 10970753270506 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 744085595 pkt, 984080080452 bytes mcast: 0 pkt, 0 bytes

9968667882 packets input, 10971596827481 bytes, 0 no buffer

Received 9400016 broadcasts, 0 runts, 0 giants, 2 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

744718114 packets output, 981254288977 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

interface Vlan70

mtu 1526

ip address x.x.x.163 255.255.255.240

no ip redirects

no ip unreachables

ip mtu 1500

load-interval 30

standby 70 ip x.x.x.161

standby 70 priority 105

standby 70 preempt

end

SWITCH_B#sh int vlan 70

Vlan70 is up, line protocol is up

Hardware is EtherSVI, address is 0015.2cb6.de00 (bia 0015.2cb6.de00)

Description:

Internet address is x.x.x.163/28

MTU 1526 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 2/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:01, output hang never

Last clearing of "show interface" counters 6d11h

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 7896000 bits/sec, 5937 packets/sec

30 second output rate 0 bits/sec, 0 packets/sec

L2 Switched: ucast: 1462911798 pkt, 665826931259 bytes - mcast: 3032920 pkt, 192202566 bytes

L3 in Switched: ucast: 3083188365 pkt, 567226593864 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 19347908 pkt, 9452365132 bytes mcast: 0 pkt, 0 bytes

3086272153 packets input, 567430451135 bytes, 0 no buffer

Received 3032938 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

19561689 packets output, 9393901802 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

Mauricio Naegele da Cruz · ‎01-28-2011

I also notice ospf flaps at the same time of the hsrp flaps (not the same vlans ...rs), i´m suspecting that the peak of traffic it´s really the problem. How can i confirm my suspicious ?

Satya Narra · ‎01-28-2011

Input queue: 0/75/112/0 (size/max/drops/flushes); Total output drops: 0

Q depth is just 75,

try hold-queue 2000 on the vlan interface. It should resolve it.

But, also go through the following documentation.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml#topic2

In case if it does not resolve the issue, we need to find out what kind of traffic is going to the CPU at the time of drops.

Have a great day.

Mauricio Naegele da Cruz · ‎02-02-2011

I´ve adjust SPT with HSRP and change the queue and solve the problem.

Thanks for the help !