cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1183
Views
0
Helpful
12
Replies

%HSRP-5-STATECHANGE: Port-channel Error - Pls Help!!

johndox
Level 1
Level 1

kindly I need your assistance on this. Having this error for the pass one month on our Distribution switch 6880x( Software (c6880x-ADVENTERPRISEK9-M), Version 15.4(1)SY1).

 

Note, CPU process is good, no stp instances, portchannel is up up. See attachment for detail

12 Replies 12

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello,

go the switch shell:

take show standby

 

moving from Active --> Speak should mean a better HSRP router is heard and the local router gives up the active role.

Verify that an active router is known for each group even if it is not the local router.

 

No STP instances running .

all interfaces on the C68800 are routed interfaces ?

 

Hope to help

Giuseppe

 

@Giuseppe Larosa 

 

kindly see Attached. 

 

And yes, to STP instances running and the sub PortChannel interfaces are routed.

Hello,

please attach as a text file the show commands or configurations

thanks.

 

Best Regards

Giuseppe

 

@Giuseppe Larosa Please check your inbox for the running-config. I had to do this because of the sensitivity of the info provided. Thanks in advance 

Hello ilenjohnx,

 

you are using very aggressive timers on multiple HSRP groups with MD5 authentication

 

Example:

standby 2009 timers msec 100 msec 300
standby 2009 priority 150
standby 2009 preempt delay minimum 60
standby 2009 authentication md5 key-chain hsrp50.2009

 

This can lead to HSRP change of states in presence of microbursts.

 

I would suggest to use less aggressive timers and to see if the network becomes more stable

something like:

standby 2009 timers msec 300 msec 950

 

Of course you should make these changes on both switches for all groups and then look at  log messages to see if the network becomes more stable.

 

Edit:

you have also CoPP Control Plane policing enabled. It would be interesting to understand in which class are HSRP hello messages classified.

 

Hope to help

Giuseppe

 

Will try this and get back to you

@Giuseppe Larosa @balaji.bandi 

 

So i have discovered that hsrp hello packets are policed on class-map: class-copp-mcast-v4-data-on-routedPort (match-any) as seen below. kindly give guide me through finding exactly with Multicast are being dropped?

 

Please, give me all the commands to use for my investigations.

 

c6880x# show policy-map control-plane

Control Plane Interface

Service-policy input: policy-default-autocopp

Hardware Counters:

class-map: class-copp-mcast-v4-data-on-routedPort (match-any)
Match: none
police :
10 pps 1 limit 1 extended limit
Earl in slot 5 :
401174895 packets
5 minute offered rate 20 pps
aggregate-forwarded 0 packets
action: drop
exceeded 401174895 packets action: drop
aggregate-forward 0 pps exceed 18 pps

Hello IlenJohnx,

HSRP hello messages should be sent to the 224.0.0.2 all routers on subnet link local multicast address.

 

The configuration of the class-map suggests that it will drop if more the 10 multicast packets are received on an interface.

With your timers of 100 msec, 10 Hello packets are sent for each HSRP group.

 

Now, CoPP is applied on the internal link between the forwarding plane and the Main CPU to protect the CPU.

 

You can see the offered  packet rate is 20 pps and not 10 pps.

 

You can solve this in two ways:

either you increase the HSRP hello timers to 300 msec so that you get 3 packets for each HSRP group per second.

Or you need to modify the class-map action so that the police-action happens at 30 pps instead of 10 pps.

 

However, the provided output show us that these packets are all dropped:

 

class-map: class-copp-mcast-v4-data-on-routedPort (match-any)
Match: none
police :
10 pps 1 limit 1 extended limit
Earl in slot 5 :
401174895 packets
5 minute offered rate 20 pps
aggregate-forwarded 0 packets
action: drop
exceeded 401174895 packets action: drop
aggregate-forward 0 pps exceed 18 pps

 

Questions:

is slot 5 the slot of the supervisor on your C6800 system ?

 

>> 401174895 packets

 

>> exceeded 401174895 packets action: drop
aggregate-forward 0 pps exceed 18 pps

 

All packets are dropped if these packets were Hello packets you should see standby unknown on all HSRP groups and no flapping of HSRP state = no state change.

 

So I am not sure that HSRP hello messages are falling in this class.

They may be dropped in part in another class of CoPP, but it is evident they are not all dropped.

 

Hope to help

Giuseppe

 

@Giuseppe Larosa  as requested, the Supervisor is in Slot 5. See below

 

What is the relevant? Please share

 

switch#show module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
5 20 6880-X-LE 16P SFP+ Multi-Rate (Active) C6880-X-LE-SUP SAL1848522Z

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
5 7c69.f6e7.ca57 to 7c69.f6e7.ca6f 1.2 15.1(02)SY01 15.4(1)SY1 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 4 C6880-X-LE-PFC SAL18453XD0 1.1 Ok

Mod Online Diag Status
---- -------------------
5 Pass

Hello,

I have asked what module was in slot 5 because it was listed in the output of the CoPP class.

 

As I have explained in my previous post all packets look like dropped in that class and this makes me to think that HSRP hello messages should not be classified / processed by this class.

However, it is possible that the HSRP hello messages are processed in another CoPP class and they can experience some drops (but not drops of all packets).

 

In other words if  in show standby you don't see standby router unknown for all groups this means that not all HSRP packets are dropped.

On the other hand if all HSRP groups have Active local standby unknown then all HSRP messages are actully dropped.

In this case either you modify the CoPP policy to police at 30 pps instead of 10 pps or you increase the HSRP timers so that less HSRP hellos are sent per second.

If you like you can post show standby brief so that we can see what is happening.

However, I have explained what to check above.

 

Hope to help

Giuseppe

 

@Giuseppe Larosa  -  pls check your inbox too for full output

 

As requested, 

 

switch# show standby brief

Interface Grp Pri P State Active Standby Virtual IP
Po50.x 100 x P Standby x.x.x.x local x.x.x.x

 

As it can be seen for all the group, all the Active standby have known IP

 

I will like to still investigate packet drops on a class map, CPU also

 

Kindly, share show commands to use for the this task. I want know the the type of  packets drop in a class map and CPU, if necessary  

 

I know about the show policy-map control-plane and show platform qos ip 

Thanks in advance

Hello John,

I have examined the log files that you have sent me.

I provide here my answers :

 

The mentioned extended ACLs are not blocking HSRP messages:

HSRP hellos are sent to destination address 224.0.0.2 all routers on subnet. (or other link local multicast for HSRP v2).

 

As I have explained before you are using timers that are too much aggressive.

There is no real need to use an HSRP timer of 100 msec in each group.

Move to 300 msec HSRP hello timers and you will solve your issue.

 

Please be aware that I am providing just an advice, if you want more assistance you should buy professional services from cisco or a cisco gold partner.

 

Hope to help

Giuseppe

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco