HSRP not working

Mukund M Deshpande · ‎09-21-2011

Hi guys ,

I have a issue here with HSRP , redundancy not working , what is happening if primary switch get reloaded traffic is not shifting on secondary one and there is a outage of 10mins around ,once primary got up all traffic found working fine via Primary switch .below are the logs after reload -

Log Buffer (4096 bytes):

SRP-5-STATECHANGE: Vlan34 Grp 34 state Listen -> Active

*Sep 14 17:47:38.491 india: %HSRP-5-STATECHANGE: Vlan41 Grp 41 state Listen -> Active

*Sep 14 17:47:38.519 india: %HSRP-5-STATECHANGE: Vlan4 Grp 4 state Listen -> Active

*Sep 14 17:47:38.663 india: %HSRP-5-STATECHANGE: Vlan54 Grp 54 state Listen -> Active

*Sep 14 17:47:38.735 india: %HSRP-5-STATECHANGE: Vlan43 Grp 43 state Listen -> Active

*Sep 14 17:47:38.767 india: %HSRP-5-STATECHANGE: Vlan11 Grp 11 state Listen -> Active

here it says vlan group state change from listen to Active where I understand normally it sud be Standby to Active .

There are 60 groups configured and all are with same priority on Primary switch -

like in primary sw->>

-------------------

!

interface Vlan6

description "Voice"

ip address 10.229.136.3 255.255.255.0

standby 6 ip 10.229.136.5

standby 6 priority 120

standby 6 preempt

!

Secondary sw-->>

-----------------

!

interface Vlan6

description "Voice"

ip address 10.229.136.4 255.255.255.0

standby 6 ip 10.229.136.5

standby 6 priority 110

standby 6 preempt

!

#show standby --->>

Vlan6 - Group 6

State is Active

1 state change, last state change 6d17h

Virtual IP address is 10.229.136.5

Active virtual MAC address is 0000.0c07.ac06

Local virtual MAC address is 0000.0c07.ac06 (v1 default)

Hello time 3 sec, hold time 10 sec

Next hello sent in 2.112 secs

Preemption enabled

Active router is local

Standby router is 10.229.136.4, priority 110 (expires in 9.728 sec)

Priority 120 (configured 120)

Group name is "hsrp-Vl6-6" (default)

Vlan6 - Group 6

State is Standby

10 state changes, last state change 6d17h

Virtual IP address is 10.229.136.5

Active virtual MAC address is 0000.0c07.ac06

Local virtual MAC address is 0000.0c07.ac06 (v1 default)

Hello time 3 sec, hold time 10 sec

Next hello sent in 1.344 secs

Preemption enabled

Active router is 10.229.136.3, priority 120 (expires in 9.936 sec)

Standby router is local

Priority 110 (configured 110)

Group name is "hsrp-Vl6-6" (default)

other details -

switch =>> WS-C4507R-E

sup=>> Sup 6L-E 10GE (X2), 1000BaseX (SFP) WS-X45-SUP6L-E

image =>>cat4500e-ipbasek9-mz.122-53.SG1.bin

I do not have any clue why hsrp did not worked at the time of reboot of primary switch and n/w got impacted .

Thanks for your reply

=================

kapathak · ‎09-21-2011

Hi Sagar,

What was the HSRP state of the secondary when the primary went down? Do we have a sh standby br or similar output from that time period?

Also, can you paste the configs of the physical interfaces connecting the two?

Mukund M Deshpande · ‎09-21-2011

Hi Kapil ,

unfortunately we do not have tht duration logs from secondary sw .here is the logs for interfaces connecting two -

from primary -

-----------------

!

interface Port-channel1

switchport

switchport trunk native vlan 32

switchport mode trunk

!

interface GigabitEthernet3/3

switchport trunk native vlan 32

switchport mode trunk

channel-protocol lacp

channel-group 1 mode active

!

interface GigabitEthernet3/4

switchport trunk native vlan 32

switchport mode trunk

channel-protocol lacp

channel-group 1 mode active

!

interface Vlan32

description "Desktop-1"

ip address 10.229.162.3 255.255.255.0

standby 32 ip 10.229.162.5

standby 32 priority 120

standby 32 preempt

!

GigabitEthernet3/3 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet Port, address is c84c.7502.06c2 (bia c84c.7502.06c2)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX

input flow-control is on, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:01, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 5537000 bits/sec, 523 packets/sec

5 minute output rate 343000 bits/sec, 354 packets/sec

129787914 packets input, 156650214823 bytes, 0 no buffer

Received 10292942 broadcasts (10278380 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

159794139 packets output, 74483070712 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

GigabitEthernet3/4 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet Port, address is c84c.7502.06c3 (bia c84c.7502.06c3)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX

input flow-control is on, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 2771000 bits/sec, 354 packets/sec

5 minute output rate 610000 bits/sec, 507 packets/sec

120720614 packets input, 89711574998 bytes, 0 no buffer

Received 463735 broadcasts (34248 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

316867312 packets output, 301234550986 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

========

from secondary sw ->>

!

interface Port-channel1

switchport

switchport trunk native vlan 32

switchport mode trunk

!

interface GigabitEthernet3/3

switchport trunk native vlan 32

switchport mode trunk

channel-protocol lacp

channel-group 1 mode active

!

interface GigabitEthernet3/4

switchport trunk native vlan 32

switchport mode trunk

channel-protocol lacp

channel-group 1 mode active

!

GigabitEthernet3/3 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet Port, address is 5475.d0c4.2082 (bia 5475.d0c4.2082)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX

input flow-control is on, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:13, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 12

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 363000 bits/sec, 378 packets/sec

5 minute output rate 5910000 bits/sec, 557 packets/sec

1011836806 packets input, 139904442207 bytes, 0 no buffer

Received 679013359 broadcasts (650678708 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

3395261351 packets output, 2495748692670 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

GigabitEthernet3/4 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet Port, address is 5475.d0c4.2083 (bia 5475.d0c4.2083)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX

input flow-control is on, output flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:23, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 14

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 718000 bits/sec, 479 packets/sec

5 minute output rate 2525000 bits/sec, 361 packets/sec

582304389 packets input, 323683240079 bytes, 0 no buffer

Received 36684102 broadcasts (1189563 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

4110092120 packets output, 4902605108078 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swaped

Thanks 4 reply

-------------------------------

dominic.caron · ‎09-21-2011

Do you have any logs on the second switch^

Limitless1801 · ‎09-21-2011

Please take a look at this paragraph from the following cisco document about troubleshooting HSRP

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a0080094afd.shtml

In order to minimize network traffic, only the active and the standby routers send periodic HSRP messages after the protocol has completed the election process. Additional routers in the HSRP group remain in the Listen state. If the active router fails, the standby router takes over as the active router. If the standby router fails or becomes the active router, another router is elected as the standby router.

It seems like you had already configured an active and standby device so that other switch was in listening state. I would probably remove the preempt from the switch with priority 110.

Can you post some debug standby events so we can see how they are talking to each other?

Thanks, RG

Mukund M Deshpande · ‎09-26-2011

Hi RG ,

here are the debug standby events -

4500-core1#debug standby events

HSRP Events debugging is on

INRNMD1L3_Core1#

*Sep 22 04:02:53.479 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.151 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:02:53.503 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.239 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:02:53.875 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.169 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:02:54.203 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.161 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:02:54.699 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.43 tgt 10.229.163.5, reply with mac 0000.0c07.ac21de

*Sep 22 04:02:55.631 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.197 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:02:56.127 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.51 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:02:56.431 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.123 tgt 10.229.155.5, reply with mac 0000.0c07.ac19b

INRNMD1L3_Core1#debug

*Sep 22 04:02:57.751 india: HSRP: Vl6 Grp 6 ARP src 10.229.136.19 tgt 10.229.136.5, reply with mac 0000.0c07.ac06

*Sep 22 04:02:58.527 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.248 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:02:59.603 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.23 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:02:59.779 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.131 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:01.627 india: HSRP: Vl20 Grp 20 ARP src 10.229.150.41 tgt 10.229.150.5, reply with mac 0000.0c07.ac14

*Sep 22 04:03:02.627 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.58 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:03.247 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.122 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:03:03.723 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.192 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:04.259 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.124 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:04.407 india: HSRP: Vl20 Grp 20 ARP src 10.229.150.100 tgt 10.229.150.5, reply with mac 0000.0c07.ac14

*Sep 22 04:03:04.735 india: HSRP: Vl14 Grp 14 ARP src 10.229.144.115 tgt 10.229.144.5, reply with mac 0000.0c07.ac0e

*Sep 22 04:03:05.147 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.129 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:05.451 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.154 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:06.015 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.155 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:06.419 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.173 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:07.475 india: HSRP: Vl20 Grp 20 ARP src 10.229.150.36 tgt 10.229.150.5, reply with mac 0000.0c07.ac14

*Sep 22 04:03:07.655 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.145 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:08.779 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.166 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:03:09.439 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.77 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:10.175 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.69 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:10.443 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.103 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:11.047 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.113 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

% Type "debug ?" for a list of subcommands

INRNMD1L3_Core1#

*Sep 22 04:03:11.735 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.176 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:03:13.203 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.121 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:03:14.379 india: HSRP: Vl24 Grp 24 ARP src 10.229.154.117 tgt 10.229.154.5, reply with mac 0000.0c07.ac18

*Sep 22 04:03:16.911 india: HSRP: Vl15 Grp 15 ARP src 10.229.145.230 tgt 10.229.145.5, reply with mac 0000.0c07.ac0f

*Sep 22 04:03:17.355 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.154 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:17.723 india: HSRP: Vl33 Allow proxy ARP, src 10.229.163.244 tgt 102.229.163.5 (soln 1) mac 0000.0c07.ac21 (grp 33)

*Sep 22 04:03:17.783 india: HSRP: Vl14 Grp 14 ARP src 10.229.144.101 tgt 10.229.144.5, reply with mac 0000.0c07.ac0e

*Sep 22 04:03:18.111 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.176 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:18.887 india: HSRP: Vl26 Grp 26 ARP src 10.229.156.180 tgt 10.229.156.5, reply with mac 0000.0c07.ac1a

*Sep 22 04:03:21.275 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.90 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:21.443 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.64 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:22.015 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.243 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:22.255 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.190 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:22.683 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.89 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:23.167 india: HSRP: Vl6 Grp 6 ARP src 10.229.136.25 tgt 10.229.136.5, reply with mac 0000.0c07.ac06

*Sep 22 04:03:25.219 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.118 tgt 10.229.155.5, reply with mac 0000.0c07.ac19

*Sep 22 04:03:25.603 india: HSRP: Vl25 Grp 25 ARP src 10.229.155.67 tgt 10.229.155.5, reply with mac 0000.0c07.ac19u

*Sep 22 04:03:26.431 india: HSRP: Vl33 Grp 33 ARP src 10.229.163.113 tgt 10.229.163.5, reply with mac 0000.0c07.ac21

*Sep 22 04:03:26.443 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.121 tgt 10.229.162.5, reply with mac 0000.0c07.ac20n c

*Sep 22 04:03:27.787 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.166 tgt 10.229.162.5, reply with mac 0000.0c07.ac2all

Limitless1801 · ‎09-26-2011

According to your logs, you have multiple IP addresses associated with the same group. For example:

*Sep 22 04:02:53.479 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.151 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:02:54.203 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.161 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

*Sep 22 04:03:03.723 india: HSRP: Vl32 Grp 32 ARP src 10.229.162.192 tgt 10.229.162.5, reply with mac 0000.0c07.ac20

10.229.162.5 is your standby ip address. You also have .151, .161 and .192 as part of VI32. If you have only two switches, why so many IP addresses?

Thanks, RG.