Solved: HSRP and BGP: secondary router keeps trying to establish BGP with ISP

Ramazan Celtek · ‎12-11-2013

Hi all,

We have two routers, running HSRP between them.

Our ISP have reported that their PE router (192.168.10.81) is getting hammered with attempts from 192.168.10.84 (our secondary router) trying to establish BGP with it.

HSRP seems to be configured correctly however am unsure if BGP is configured correctly to use the HSRP IP for the BGP relationship.

Can someone please assist with diagnosing the problem, that is why does our secondary router continue to attempt a BGP relationship with ISP PE router?

Config & status outputs are below:

Many thanks in advance.

Rama

**************************************

Primary router MELRTRW001:

interface GigabitEthernet0/1

bandwidth 76800

ip address 192.168.10.83 255.255.255.248

no ip proxy-arp

ip nbar protocol-discovery

max-reserved-bandwidth 100

service-policy output 75Mbs_WAN_Service

ip route-cache flow

load-interval 30

...

standby 2 ip 192.168.10.82

standby 2 priority 110

standby 2 preempt

standby 2 authentication cpaaust

standby 2 track GigabitEthernet0/0 50

Secondary MELRTRW002:

interface GigabitEthernet0/1

bandwidth 76800

ip address 192.168.10.84 255.255.255.248

no ip proxy-arp

max-reserved-bandwidth 100

service-policy output 75Mbs_WAN_Service

...

standby 2 ip 192.168.10.82

standby 2 priority 90

standby 2 authentication cpaaust

standby 2 track GigabitEthernet0/0 50

BGP Neighbour status from primary router:

melrtrw001#sh ip bg neighbors 192.168.10.81

BGP neighbor is 192.168.10.81, remote AS XXXX, external link

BGP version 4, remote router ID 192.168.10.81

BGP state = Established, up for 4w4d

Last read 00:00:00, hold time is 90, keepalive interval is 30 seconds

Neighbor capabilities:

Route refresh: advertised and received(old & new)

Address family BGP IPv4: advertised and received

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 3 2

Notifications: 0 0

Updates: 2 93

Keepalives: 94525 96565

Route Refresh: 0 0

Total: 94566 96660

Default minimum time between advertisement runs is 30 seconds

For address family: BGP IPv4

BGP table version 724, neighbor version 724/0

Output queue size : 0

Index 1, Offset 0, Mask 0x2

1 update-group member

Inbound soft reconfiguration allowed

Default information originate, default sent

Inbound path policy configured

Outbound path policy configured

Route map for incoming advertisements is IMPORT-POLICY

Route map for outgoing advertisements is Routes_to_ASXXXX

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 34 64 (Consumes 3024 bytes)

Prefixes Total: 48 165

Implicit Withdraw: 0 18

Explicit Withdraw: 14 83

Used as bestpath: n/a 55

Used as multipath: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

route-map: 166 0

Bestpath from this peer: 9 n/a

Total: 175 0

Number of NLRIs in the update sent: max 0, min 0

Connections established 2; dropped 1

Last reset 4w4d, due to Peer closed the session

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled

Local host: 192.168.10.82, Local port: 179

Foreign host: 192.168.10.81, Foreign port: 53013

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0xA90582B4):

Timer Starts Wakeups Next

Retrans 94530 0 0x0

TimeWait 0 0 0x0

AckHold 96607 93701 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 0 0 0x0

DeadWait 0 0 0x0

iss: 938153014 snduna: 939950022 sndnxt: 939950022 sndwnd: 16384

irs: 3031418952 rcvnxt: 3033258450 rcvwnd: 16175 delrcvwnd: 209

SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms

Flags: passive open, nagle, gen tcbs

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 190810 (out of order: 0), with data: 96607, total data bytes: 1839497

Sent: 189490 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 94531, total data bytes: 1797007

BGP Neighbour status from secondary router:

melrtrw002#sh ip bgp neighbors 192.168.10.81

BGP neighbor is 192.168.10.81, remote AS XXXX, external link

BGP version 4, remote router ID 0.0.0.0

BGP state = Active

Last read 00:00:13, hold time is 180, keepalive interval is 60 seconds

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 64393 0

Notifications: 0 0

Updates: 0 0

Keepalives: 0 0

Route Refresh: 0 0

Total: 64393 0

Default minimum time between advertisement runs is 30 seconds

For address family: BGP IPv4

BGP table version 605, neighbor version 0/0

Output queue size : 0

Index 1, Offset 0, Mask 0x2

1 update-group member

Inbound soft reconfiguration allowed

Default information originate, default not sent

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 0 0

Prefixes Total: 0 0

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 0

Used as multipath: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

Total: 0 0

Number of NLRIs in the update sent: max 0, min 0

Connections established 0; dropped 0

Last reset never

No active TCP connection

Debug IP BGP logs on secondary router below:

Dec 11 23:53:19: BGP: Applying map to find origin for xxx.xxx.xxx.xxx/30

...

Dec 11 23:53:23: BGP: 192.168.10.81 went from Idle to Active

Dec 11 23:53:23: BGP: 192.168.10.81 open active, delay 21059ms

Dec 11 23:53:44: BGP: 192.168.10.81 open active, local address 192.168.10.84

Dec 11 23:53:44: BGP: 192.168.10.81 went from Active to OpenSent

Dec 11 23:53:44: BGP: 192.168.10.81 sending OPEN, version 4, my as: XXXXX, holdtime 180 seconds

Dec 11 23:53:44: BGP: 192.168.10.81 send message type 1, length (incl. header) 45

Dec 11 23:53:45: BGP: 192.168.10.81 remote close, state CLOSEWAIT

Dec 11 23:53:45: BGP: 192.168.10.81 -reset the session

Dec 11 23:53:46: BGPNSF state: 192.168.10.81 went from nsf_not_active to nsf_not_active

Dec 11 23:53:46: BGP: 192.168.10.81 went from OpenSent to Idle

Dec 11 23:53:46: BGP: 192.168.10.81 closing

Dec 11 23:54:06: BGP: 192.168.10.81 went from Idle to Active

Dec 11 23:54:06: BGP: 192.168.10.81 open active, delay 24058ms

HSRP status on primary router:

melrtrw001# sh standby GigabitEthernet0/1

GigabitEthernet0/1 - Group 2

State is Active

4 state changes, last state change 4w4d

Virtual IP address is 192.168.10.82

Active virtual MAC address is 0000.0c07.ac02

Local virtual MAC address is 0000.0c07.ac02 (v1 default)

Hello time 3 sec, hold time 10 sec

Next hello sent in 1.852 secs

Authentication text "cpaaust"

Preemption enabled

Active router is local

Standby router is 192.168.10.84, priority 90 (expires in 8.132 sec)

Priority 110 (configured 110)

Track interface GigabitEthernet0/0 state Up decrement 50

IP redundancy name is "hsrp-Gi0/1-2" (default)

melrtrw001#

HSRP status on secondary router:

melrtrw002# sh standby GigabitEthernet0/1

GigabitEthernet0/1 - Group 2

State is Standby

4 state changes, last state change 4w4d

Virtual IP address is 192.168.10.82

Active virtual MAC address is 0000.0c07.ac02

Local virtual MAC address is 0000.0c07.ac02 (v1 default)

Hello time 3 sec, hold time 10 sec

Next hello sent in 0.808 secs

Authentication text "cpaaust"

Preemption disabled

Active router is 192.168.10.83, priority 110 (expires in 8.528 sec)

Standby router is local

Priority 90 (configured 90)

Track interface GigabitEthernet0/0 state Up decrement 50

IP redundancy name is "hsrp-Gi0/1-2" (default)

Julio Carvajal · ‎12-11-2013

Hello Ramazan,

Based on the configuration shown so far we can see that HSRP is doing it's job but the problem is with the BGP relationship as the Secondary unit is trying to bring up a TCP session from the Virtual HSRP address which is already up.

This is the expected behavior as with HSRP you can always source traffic from an interface and not for a specific interface.

That being said can you configure the following on the HSRP routers and let us know if it helps

neig x.x.x.x transport connection-mode passive

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

View solution in original post

Julio Carvajal · ‎12-11-2013

Hello Ramazan,

Based on the configuration shown so far we can see that HSRP is doing it's job but the problem is with the BGP relationship as the Secondary unit is trying to bring up a TCP session from the Virtual HSRP address which is already up.

This is the expected behavior as with HSRP you can always source traffic from an interface and not for a specific interface.

That being said can you configure the following on the HSRP routers and let us know if it helps

neig x.x.x.x transport connection-mode passive

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

lmediavilla · ‎12-12-2013

The solution by Julio is great routers will passive wait for bgp session to establish from the ISP to your active router.

cheers

Julio Carvajal · ‎12-12-2013

Hello,

Thanks for the comments

This should fix the problem.

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Ramazan Celtek · ‎12-12-2013

Hi Julio,

Thanks for the reply. It's very much appreciated.

This sounds like a great way to fix it, though I'm sure Milan's suggestion would also work.

I'm really looking for the least disruptive method with minimal changes at this point. Organisation & carrier embargo's will be in place soon too.

I can apply the following command to my secondary router.

neig x.x.x.x transport connection-mode passive

Will i then need to apply the equivalent 'active' command to my primary router? I would think not as our ISP only has a BGP relationship configured with our primary router IP & is working fine.

Thanks again

Rama

Jon Marshall · ‎12-12-2013

Rama

Your BGP peering with the ISP is only working because it is passive ie. from your output -

Local host: 192.168.10.82, Local port: 179

Foreign host: 192.168.10.81, Foreign port: 53013

this shows that the ISP router started the connection to your primary router. Using HSRP as the neighbor address only works when the neighbor initiates the connection. When your router tries to start a connection it uses it's real IP not the virtual one and so the neighborship cannot be formed. That is one of the drawbacks to using the HSRP virtual address.

So there is no point in using active, in fact if you did you would never get a peering on the primary router. You need to apply the command Julio supplied on both routers because if HSRP has to failover then your current primary router would then be sending requests to the ISP router and you would have the same problem you have now.

Actually i think, although i cannot test it, it could be worse than this. Your current primary (R1) fails. Your standby router (R2) takes control of the virtual IP/virtual mac address. Your standby router has no BGP session and cannot initiate one because you are using the HSRP virtual IP. So you have to wait for the ISP router (SP1) to realise the connection is dead and tear it down. But what happens if R1 failed and rebooted itself. While it was rebooting SP1s holdtime expired, it tears down the connection and tries to open a new one. A new connection is established with R2. R1 finishes rebooting, comes back online and takes back control of the virtual IP/virtual mac. The keepalives sent by SP1 are to the virtual IP so they go to R1. But R1 has no BGP session open. R2 does not get any keepalives so it thinks the connection is down. In addition because R2 no longer controls the virtual IP i am not sure whether it could send any keepalives with that as the source IP. Either way the connection is broken and you have to wait again until SP1 times that connection out and switches back to R1.

I cannot say for sure the above would happen, and would be interested to hear other's opinions on this, but i can't see why it wouldn't.

As i say, to answer your specific question the same command needs to be applied on both routers. I do not know whether you can apply that command on your primary router without it resetting the peering because i have never done it before. It may be, because the connection is passive anyway, that it will not reset the connection but i can't say for sure.

Perhaps Julio can answer.

Jon

Julio Carvajal · ‎12-12-2013

Hello All,

No, the session will not be torn down but of course changes will not be present till you clear the BGP relationship.

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Jon Marshall · ‎12-13-2013

Julio

Why is the session not torn down when the HSRP virtual IP changes to the backup router ?

Can you explain how it would work from the example i described. If R1 fails, R2 takes control of the virtual IP. Keepalives are still being sent by SP1 to the virtual IP but these go to R2 now. R2 has no BGP session open and cannot initiate one, it can only wait for SP1 to start the connection.

So i would have thought that SP1 would have to tear down the session to R1 because it was not getting any keepalives and start a new session with R2.

I'm not saying i am right, just want to understand how the BGP session is switched to R2 without tearing down the old one and starting a new one.

Jon

milan.kulik · ‎12-13-2013

Hi Jon,

this is exactly what I also think.

I made a quick test yesterday in my lab to prove and the BGP session was restarted when HSRP swapped!

There might be also another issue - a different BGP router ID (unless configured the same on both routers)!

@Julio:

Are you using some trick to keep the BGP session up?

Best regards,

Milan

Jon Marshall · ‎12-13-2013

Hi Milan

In your lab setup when HSRP swapped how long did it take for the new BGP session to start ? Did you have to wait for the holdtime to expire before it tried to open up a new connection to the HSRP virtual IP ?

I don't have anything to test with but i am assuming that when R2 starts receiving keepalives from SP1 it simply ignores them because it does not have a TCP session already open.

Jon

Julio Carvajal · ‎12-13-2013

Hello Jonm

I tought that we were talking about when we use the passive command, not when we changed the BGP neigbhor IP address.

To what you are referring the answer is yes, should be restarted.

Rate all of the helpful posts!!!

Regards,

Jcarvaja

Follow me on http://laguiadelnetworking.com

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Jon Marshall · ‎12-13-2013

Julio

Thanks, that makes more sense now

Jon

Ramazan Celtek · ‎12-17-2013

Thanks guys

So to confirm, i have currently only made added the passive command to my secondary router which has stopped the ISP from complaining.

However reading the posts, i should be adding the same command to the primary router.

1. Will this cause an issue with the present bgp relationship?

2. Will i need to do a soft clear for it to take effect? and will this cause the bgp relationship to restart?

Rama

milan.kulik · ‎12-17-2013

Hi Rama,

IMHO,

a) You would need your ISP to configure

nei ... transport connection-mode active

on his router to be 100% sure this solution works fast under any conditions.

b) I still think this is not an optimal design.

As your BGP session will get restarted any time your HSRP flaps for any reason.

Why don't you configure a standard primary/backup solution with both your routers peering to the ISP router(s)?

Best regards,

Milan

Jon Marshall · ‎12-18-2013

Rama

1) According to Julio, not it won't.

2) You don't need it to take effect because your connection from the active router is already passive so just leave it.

That said, i agree with Milan. It is not an optimal design in terms of redundancy. In addition to HSRP flaps if you needed to take the primary down for maintenance it would switch to the backup. But as soon as you brought the primary back up you would lose BGP again until it switched over.

With 2 separate peerings this would not be an issue.

Jon