HSRP Causes Terminal Lag

SniffingPackets · ‎10-26-2022

Hi All,

We are having a weird issue and while we wait on Cisco TAC to evaluate I thought of polling the community.

In this set up we have two ISR router in a HSRP group. They are connected to two Catalyst 9300 with a port channel between them.

We configured an SVI on both switches and added it to the HSRP group. Immediately after adding the HSRP you can feel the terminal getting laggy and slow.

We are connected to the switches via SSH on the management port.

We also had an event where one of the switch became active, but there were other routers with higher priority which was active. During this time we couldn’t poll the switch in question with SNMP and ICMP was not responding (management port). The device did not reboot.

MHM Cisco World · ‎10-26-2022

do you check the CPU in both SW?

SniffingPackets · ‎10-26-2022

The CPU looks normal. When we add the HSRP config and the console is laggy the CPU was below 3%. When the switch stop responding to SNMP and PING (HSRP also became rogue by going Active) i could not pull the CPU. and because SNMP stopped working we have a gap in our NMS

MHM Cisco World · ‎10-26-2022

what I think is HSRP use multicast to exchange hello, this multicast can lead to storm if you have L2 issue, this storm stop SW from respond to all control traffic include SSH.
what we need to see if you not clear interface counter is
show interface
check multicast count is it rapidly increase ?
check multicast/broadcast count is to high?

SniffingPackets · ‎10-26-2022

I was going down the same line of thinking. but the increase in multicast packets is inline with the HSRP hello timer.

The the last poll we have before the switch stop responding and the first poll after the device looks to be normal - there is no deviation/spike in that value. mcast packets, broadcast packets cpu all looks fine.

MHM Cisco World · ‎10-26-2022

Switch#show platform software fed switch active punt cause summary

Switch#show platform software fed switch active cpu-interface

please share the output of above command

SniffingPackets · ‎10-26-2022

switch2#show platform software fed switch active punt cause summary
Statistics for all causes

Cause  Cause Info                      Rcvd                 Dropped
------------------------------------------------------------------------------
7      ARP request or response         2465757              0
11     For-us data                     1790165              0
21     RP<->QFP keepalive              1101603              0
24     Glean adjacency                 3                    0
55     For-us control                  90998                0
60     IP subnet or broadcast packet   32                   0
96     Layer2 control protocols        1370105              0
------------------------------------------------------------------------------

switch2#show platform software fed switch active cpu-interface
queue                      retrieved   dropped     invalid     hol-block
-------------------------------------------------------------------------
Routing Protocol           90998       0           0           0
L2 Protocol                269200      0           0           0
sw forwarding              3           0           0           0
broadcast                  32          0           0           0
icmp gen                   0           0           0           0
icmp redirect              0           0           0           0
logging                    0           0           0           0
rpf-fail                   0           0           0           0
DOT1X authentication       0           0           0           0
Forus Traffic              1790165     0           0           0
Forus Resolution           2465757     0           0           0
Inter FED                  0           0           0           0
L2 LVX control             0           0           0           0
EWLC control               0           0           0           0
EWLC data                  0           0           0           0
L2 LVX data                0           0           0           0
Openflow                   0           0           0           0
Topology control           1100921     0           0           0
Proto snooping             0           0           0           0
DHCP snooping              0           0           0           0
Transit Traffic            0           0           0           0
Multi End station          0           0           0           0
Webauth                    0           0           0           0
High rate app              0           0           0           0
Exception                  0           0           0           0
System Critical            0           0           0           0
NFL sampled data           0           0           0           0
Low latency                0           0           0           0
EGR exception              0           0           0           0
Stackwise Virtual OOB      0           0           0           0
Multicast data             0           0           0           0
Gold packet                0           0           0           0

MHM Cisco World · ‎10-26-2022

thanks do same show and share here
we must see not rapidly increase number

SniffingPackets · ‎10-27-2022

Second show

switch2#show platform software fed switch active punt cause summary
Statistics for all causes

Cause  Cause Info                      Rcvd                 Dropped
------------------------------------------------------------------------------
7      ARP request or response         2465757              0
11     For-us data                     1790165              0
21     RP<->QFP keepalive              1141519              0
24     Glean adjacency                 3                    0
55     For-us control                  90998                0
60     IP subnet or broadcast packet   32                   0
96     Layer2 control protocols        1419775              0
------------------------------------------------------------------------------

switch2#show platform software fed switch active cpu-interface
queue                      retrieved   dropped     invalid     hol-block
-------------------------------------------------------------------------
Routing Protocol           90998       0           0           0
L2 Protocol                278957      0           0           0
sw forwarding              3           0           0           0
broadcast                  32          0           0           0
icmp gen                   0           0           0           0
icmp redirect              0           0           0           0
logging                    0           0           0           0
rpf-fail                   0           0           0           0
DOT1X authentication       0           0           0           0
Forus Traffic              1790165     0           0           0
Forus Resolution           2465757     0           0           0
Inter FED                  0           0           0           0
L2 LVX control             0           0           0           0
EWLC control               0           0           0           0
EWLC data                  0           0           0           0
L2 LVX data                0           0           0           0
Openflow                   0           0           0           0
Topology control           1140825     0           0           0
Proto snooping             0           0           0           0
DHCP snooping              0           0           0           0
Transit Traffic            0           0           0           0
Multi End station          0           0           0           0
Webauth                    0           0           0           0
High rate app              0           0           0           0
Exception                  0           0           0           0
System Critical            0           0           0           0
NFL sampled data           0           0           0           0
Low latency                0           0           0           0
EGR exception              0           0           0           0
Stackwise Virtual OOB      0           0           0           0
Multicast data             0           0           0           0
Gold packet                0           0           0           0

MHM Cisco World · ‎10-27-2022

waiting second times of show
and also
can you do
show standby
check if the mac address is same in both SW
one more point
can you confirm that you use different management interface IP in both SW??

SniffingPackets · ‎10-27-2022

I had to disable the HSRP. because it kept flapping and causing an issue.

Vlan200 - Group 0
  State is Init (interface down)
    37 state changes, last state change 1d13h
  Virtual IP address is 111.111.111.111
  Active virtual MAC address is unknown (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac00 (v1 default)
  Hello time 3 sec, hold time 10 sec
  Authentication MD5, key-string
  Preemption enabled, delay min 120 secs
  Active router is unknown
  Standby router is unknown
  Priority 40 (configured 40)
  Group name is "hsrp-Vl200-0" (default)

Mac address if different for the SVI and management interface.

I have distinct IP addresses for both mgmt interface of the switch.

MHM Cisco World · ‎10-27-2022

Vlan200 - Group 0

State is Init (interface down)

37 state changes, last state change 1d13h

switch2#show platform software fed switch active cpu-interface

queue retrieved dropped invalid hol-block

-------------------------------------------------------------------------

Routing Protocol 90998 0 0 0

L2 Protocol 278957 0 0 0

switch2#show platform software fed switch active cpu-interface

queue retrieved dropped invalid hol-block

-------------------------------------------------------------------------

Routing Protocol 90998 0 0 0

L2 Protocol 269200 0 0 0

the state change 37 that I think is large number,
also the L2 protocol is increase 10000 within small period (I compare all count for both),
L2 protocol high is usually issue in STP,
can you show spanning-tree VLAN x
check the TCN is number is high also ??

SniffingPackets · ‎10-27-2022

I agree 37 state change is high. it was flapping when we had the issue.

STP looks good to me. It was the first thing i changed. All ports are forwarding and every device connected is some time of host.

switch2#show spanning-tree vlan 100

VLAN0200
  Spanning tree enabled protocol rstp
  Root ID    Priority    32868
             Address     cc7f.7649.xxxx
             Cost        1000
             Port        3049 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    45156  (priority 45056 sys-id-ext 100)
             Address     cc7f.7653.yyyy
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/5             Desg FWD 20000     128.5    P2p Edge
Gi1/0/9             Desg FWD 20000     128.9    P2p Edge
Te1/1/1             Desg FWD 2000      128.29   P2p Edge
Po1                 Root FWD 1000      128.3049 P2p

 VLAN0200 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 45056, sysid 100, address cc7f.7653.yyyy
  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
  Current root has priority 32868, address cc7f.7649.xxxx
  Root port is 3049 (Port-channel1), cost of root path is 1000
  Topology change flag not set, detected flag not set
  Number of topology changes 1 last change occurred 3w5d ago
          from Port-channel1
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0, aging 300

switch2#   show spanning-tree detail | in ieee|from|occur|is exec
 VLAN0001 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 3 last change occurred 3w5d ago
          from GigabitEthernet1/0/9
 VLAN0200 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 1 last change occurred 3w5d ago
          from Port-channel1

MHM Cisco World · ‎10-27-2022

OK,
let follow the packet from interface to CPU,
show controller ethernet-controller <interface>
is the first point that we must check to see the multicast packet is receive in interface or not

then the interface will forward this packet to CPU Queue,
the name of Queue is Routing-Control and I see it counter increase BUT
to make sure that CoPP not drop HSRP packet in Queue we need

show platform hardware fed [switch] active qos queue state internal cpu policer

and check the Routing Control Queue Drop.

hope this time we get the issue here

SniffingPackets · ‎10-28-2022

switch2#show controller ethernet-controller TenGigabitEthernet 1/1/8
Transmit                  TenGigabitEthernet1/1/8               Receive
 805954707854 Total bytes                109382880761 Total bytes
   4093968189 Unicast frames                 92964166 Unicast frames
 803008015058 Unicast bytes              109268467789 Unicast bytes
      1082579 Multicast frames                 979166 Multicast frames
    118115496 Multicast bytes               110517388 Multicast bytes
     41596725 Broadcast frames                  57288 Broadcast frames
   2828577300 Broadcast bytes                 3895584 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames

switch2#show controller ethernet-controller TenGigabitEthernet 1/1/7
Transmit                  TenGigabitEthernet1/1/7               Receive
6674357275268 Total bytes               2236249893379 Total bytes
   9864432026 Unicast frames               2785842928 Unicast frames
6674285784824 Unicast bytes             2235898708503 Unicast bytes
       127757 Multicast frames                1328379 Multicast frames
     25231260 Multicast bytes               112206556 Multicast bytes
       680129 Broadcast frames                3514372 Broadcast frames
     46259184 Broadcast bytes               238978320 Broadcast bytes

switch2#show controller ethernet-controller GigabitEthernet 1/0/5
Transmit                  GigabitEthernet1/0/5          Receive
 539558906493 Total bytes               6725418220467 Total bytes
    473261415 Unicast frames              12991265633 Unicast frames
 539107392320 Unicast bytes             6722648050740 Unicast bytes
      2110203 Multicast frames                1003696 Multicast frames
    180400593 Multicast bytes               107780671 Multicast bytes
      4235971 Broadcast frames               41599829 Broadcast frames
    271113580 Broadcast bytes              2662389056 Broadcast bytes

All the devices in the path show multicast packets. I can't pull this data for SVI

switch2#show platform hardware fed switch active qos queue stats internal cpu policer

                         CPU Queue Statistics
============================================================================================
                                              (default) (set)     Queue        Queue
QId PlcIdx  Queue Name                Enabled   Rate     Rate      Drop(Bytes)  Drop(Frames)
--------------------------------------------------------------------------------------------
0    11     DOT1X Auth                  Yes     1000      1000     0            0
1    1      L2 Control                  Yes     2000      2000     0            0
2    14     Forus traffic               Yes     4000      4000     4223501      5320
3    0      ICMP GEN                    Yes     600       600      0            0
4    2      Routing Control             Yes     5400      5400     0            0
5    14     Forus Address resolution    Yes     4000      4000     3136         49
6    0      ICMP Redirect               Yes     600       600      0            0
7    16     Inter FED Traffic           Yes     2000      2000     0            0
8    4      L2 LVX Cont Pack            Yes     1000      1000     0            0
9    19     EWLC Control                Yes     13000     13000    0            0
10   16     EWLC Data                   Yes     2000      2000     0            0
11   13     L2 LVX Data Pack            Yes     1000      1000     0            0
12   0      BROADCAST                   Yes     600       600      0            0
13   10     Openflow                    Yes     200       200      0            0
14   13     Sw forwarding               Yes     1000      1000     0            0
15   8      Topology Control            Yes     13000     13000    0            0
16   12     Proto Snooping              Yes     2000      2000     0            0
17   6      DHCP Snooping               Yes     400       400      0            0
18   13     Transit Traffic             Yes     1000      1000     0            0
19   10     RPF Failed                  Yes     200       200      0            0
20   15     MCAST END STATION           Yes     2000      2000     0            0
21   13     LOGGING                     Yes     1000      1000     0            0
22   7      Punt Webauth                Yes     1000      1000     0            0
23   18     High Rate App               Yes     13000     13000    0            0
24   10     Exception                   Yes     200       200      0            0
25   3      System Critical             Yes     1000      1000     0            0
26   10     NFL SAMPLED DATA            Yes     200       200      0            0
27   2      Low Latency                 Yes     5400      5400     0            0
28   10     EGR Exception               Yes     200       200      0            0
29   5      Stackwise Virtual OOB       Yes     8000      8000     0            0
30   9      MCAST Data                  Yes     400       400      0            0
31   3      Gold Pkt                    Yes     1000      1000     0            0