Does not contain an active HSRP group - Speed Test

[garrybe]

I am using DMVPN and MPLS. If I run a speed test (like speedtest.com) from a local pc, HSRP flips seemingly to the MPLS very briefly. It does this as soon as the upload test begins. Several different speed testing websites have been used with the same effect:

DMVPN (terminal monitor):

007870: Jan 11 13:40:42.718 EST: %HSRP-5-STATECHANGE: GigabitEthernet0/0.5 Grp 225 state Standby -> Active

007872: Jan 11 13:40:47.886 EST: %HSRP-5-STATECHANGE: GigabitEthernet0/0.5 Grp 225 state Active -> Speak

007873: Jan 11 13:40:59.214 EST: %HSRP-5-STATECHANGE: GigabitEthernet0/0.5 Grp 225 state Speak -> Standby

The MPLS does not report anything with just terminal monitor enabled on both routers.

------

I then issued "debug standby" on the DMVPN and MPLS and ran the speed test again. The DMVPN returned the following (.251=dmvpn, .252=mpls):

007784: Jan 11 13:03:24.551 EST: HSRP: Gi0/0.5 Grp 225 Hello out 192.168.225.251 Standby pri 100 vIP 192.168.225.250
007785: Jan 11 13:03:24.815 EST: HSRP: 192.168.125.252 does not contain an active HSRP group
007786: Jan 11 13:03:24.915 EST: HSRP: Gi0/0.10 Grp 10 Hello in 10.10.125.252 Standby pri 100 vIP 10.10.125.250
007787: Jan 11 13:03:25.335 EST: HSRP: Gi0/0.5 Grp 225 Hello in 192.168.225.252 Active pri 120 vIP 192.168.225.250
007788: Jan 11 13:03:25.367 EST: HSRP: Gi0/0.10 Grp 10 Hello out 10.10.125.251 Active pri 120 vIP 10.10.125.250
007789: Jan 11 13:03:25.463 EST: HSRP: Gi0/0.1 Grp 0 Hello in 192.168.125.252 Standby pri 100 vIP 192.168.125.250

The MPLS returns:

Jan 11 13:52:48.537 EST: HSRP: Gi0/0/0.12 Grp 12 Hello out 10.12.125.252 Standby pri 100 vIP 10.12.125.250
Jan 11 13:52:48.649 EST: HSRP: Gi0/0/0.1 Grp 0 Hello out 192.168.125.252 Standby pri 100 vIP 192.168.125.250
Jan 11 13:52:49.131 EST: HSRP: Gi0/0/0.10 Grp 10 Hello out 10.10.125.252 Standby pri 100 vIP 10.10.125.250
Jan 11 13:52:49.595 EST: HSRP: Gi0/0/0.1 Grp 0 Hello in 192.168.125.251 Active pri 130 vIP 192.168.125.250
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Interface adv in, Passive, active 0, passive 1, from 192.168.225.251
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Grp 225 Resign in 192.168.225.251 Speak pri 100 vIP 192.168.225.250
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Grp 225 Active: i/Resign rcvd (100/192.168.225.251)
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Grp 225 Coup out 192.168.225.252 Active pri 120 vIP 192.168.225.250
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Grp 225 Hello out 192.168.225.252 Active pri 120 vIP 192.168.225.250
Jan 11 13:52:49.598 EST: HSRP: Gi0/0/0.5 Grp 225 Hello in 192.168.225.251 Speak pri 100 vIP 192.168.225.250
Jan 11 13:52:49.607 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 192.168.125.251 (GigabitEthernet0/0/0.1) is down: Interface PEER-TERMINATION received
Jan 11 13:52:49.611 EST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 192.168.125.251 (GigabitEthernet0/0/0.1) is up: new adjacency

So for 0/0.5, it seems that the DMVPN for some reason briefly tries to take over as active but then the MPLS sends a coup once it receives the lower priority active hello and the issue corrects itself. 

My question is what could be causing the DMVPN to act this way? Its as if something about the speed tests trigger this to happen.

[balaji.bandi]

you need to provide the what device model you using what code running, also some config example (high level network diagram for us to understand)

as per the output, seems to be when you do the test, the Links get saturated that where this flips happens., since HSRP peers are not able to see each other.

you need to check that time of testing.,

process CPU ?

interface load ?

[garrybe]

The DMVPN is a C3900-SPE100/K9 (IOS:Version 15.7(3)M8) and the MPLS is a C8300-1N1S-4T2X (Cisco IOS XE Software, Version 17.03.04a). Essentially, this a branch site and I'm at the HQ. The speed tests are being ran on a computer at the branch that I remote to. As you see in the above screenshots, the DMVPN is the active router for all HSRP groups except G0/0.5 (group 0) as that is local traffic. The DMVPN has preemption enabled on all HSRP groups except for G0/0.5  (group 0). The MPLS is the opposite, preemption disabled on all groups except for group 0.

Below is the interface load on the DMVPN shortly after running the speed test:

Below is 'show cpu proc hist' shortly after running the speed test:

[balaji.bandi]

is the issue with all the HSRP VIP ? i do see CPU hitting 100% when you run the test ?

[MHM Cisco World]

you run MPLSoDMVPN that OK, but what is HSRP, are you have two hub ? 

[garrybe]

This is a phase 1 dmvpn, 1 hub router. There are several HSRP groups that use tracks some of which ping internal ips and others external ips. I have seend all the tracks go down when running the speed test but the internal ones always go down. It seems like the DMVPN thinks it loses connection entirely with the MPLS briefly but I don't understand why.

[Georg Pauwen]

Hello,

since your DUAL is going down, it could be that the speedtest uses the entire bandwidth and starves any EIGRP traffic. It used to be that by default, EIGRP would reserve 50% of the available bandwidth, but you might want to set this to a different value on your GigabitEthernet0/0/0.1 interface:

ip bandwidth-percent eigrp 10 25

[garrybe]

Here's the show int on g0/1 on the DMVPN. There are a lot of 'Pause Outputs' and this number increments when I run the speed test to cause the issue:

5 minute input rate 114000 bits/sec, 10 packets/sec
5 minute output rate 63000 bits/sec, 10 packets/sec
17114603 packets input, 19425283416 bytes, 0 no buffer
Received 295141 broadcasts (0 IP multicasts)
0 runts, 0 giants, 166 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
12347344 packets output, 5896018498 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
24224 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier, 25433401 pause output
0 output buffer failures, 0 output buffers swapped out

I applied the command first just to the DMVPN and then just to the MPLS and finally to both but it didn't seem to make any difference in any case.

We also have QoS for the following:

class-map match-any NET-CTRL-MGMT-OUT
match dscp cs2
match dscp cs6

class NET-CTRL-MGMT-OUT
bandwidth remaining percent 5
set dscp cs6

class-map match-all NETWORK-MANAGEMENT-IN
match dscp cs2
match dscp cs6

class NETWORK-MANAGEMENT-IN
set dscp cs2

class-map match-all NETWORK-CONTROL-IN
match dscp cs2
match dscp cs6

class NETWORK-CONTROL-IN
set dscp cs6

I tried to remove the 'match dscp cs6' on NETWORK-CONTROL-IN and NET-CTRL-MGMT-OUT (not at the same time) with no change either.