BGP Issue with IPSEC Tunnels

[John Gentle]

I have an issue with IPSEC tunnels and my BGP connection.  I am going to try to lay out the topology as best as possible, so here goes. Please bear with me as a bit complicated and i have run in to something i can't figure out.

I have 2x ISPs that come in to a stacked 3750 switch on seperate VLANs (1501 and 1502).  From a 2900 is connected to the 3750's and providing BGP for my IPs.  Both interfaces feed back to the 3750.  Then from the 3750 i have core routers (in this case I am referring to a 3845) that provides internal NAT and IPSEC tunnels out to my remote locations.

In the 3750 the Vlan interface for each ISP has one of their IPs from the /29 they give me for BGP assigned to it so i can gain access remotely.

The core router has ip route 0.0.0.0 0.0.0.0 x.x.x.1 which is the 2900 router providing BGP and the x.x.x.1 is one of my IPs in my range (ie not provided by the ISP)

The edge router providing BGP has:

ip route a.a.a..201 255.255.255.255 GigabitEthernet0/0.1501  (.201 is the ISPs gateway)
ip route y.y.y.137 255.255.255.255 GigabitEthernet0/0.1502  (same as above, .137 is the ISPs gateway)

Here is where the problem comes. The 3750 stack for reasons unknown to me MUST have for the IPSEC traffic to work:

ip route 0.0.0.0 0.0.0.0 a.a.a.201

ip route 0.0.0.0 0.0.0.0 y.y.y.137 secondary

If i remove those route statements from the 3750 IPSEC traffic fails, however other traffic that is simply NAT seems to be fine - both in and out.  The problem with that is if the route that isn't secondary goes down then everything dies off until i reverse it.  Aside from that basically everything is going out the a.a.a.201 ISP regardless of AS routes which is problematic if one gets extremely loaded.

There doesn't seem to be an issue with BGP per se because i can check looking glasses and see my routes advertised properly.

My question and problem is why do i have to have the ip route statements on the 3750 stack?  I am quite sure you will need additional info to help me, so thanks in advance for any insight you can offer.

Thanks in advance for any assistance!!

John

[Georg Pauwen]

Hello John,

is this your setup ? 

Remote locations (IPSec/NAT) <--> 3845 --> 3750 Stack --> 2900 --> ISP1/ISP2 ?

[John Gentle]

Not exactly.  Remote locations (IPSec/NAT) <-ISP1/2 Internet-> 3750 stack ---> 2900 (BGP) ---> 3750 stack ---> 3845

[Georg Pauwen]

John,

I have looked at the configs. Removing the default routes on the 3750 would just leave the connected routes in the routing table. Can you issue a 'show ip route' on the 3750 for both the IPSec traffic and the NAT traffic ? And post the output ?

[John Gentle]

I should add one small correction on the 3750.  The actual route statements are:

ip route 0.0.0.0 0.0.0.0 a.a.a.201
ip route 0.0.0.0 0.0.0.0 y.y.y.137 2

I incorrectly had y.y.y.137 secondary but when i just put it back in there to generate the "show ip route" i realized my mistake.  None the less, from the 3750 with both ISP1 and ISP2 connected and all BGP tables on the 2900 updated this is what i get with a sh ip rou

at1-wanstaack-1# sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is a.a.a.201 to network 0.0.0.0

     a.0.0.0/29 is subnetted, 1 subnets
C       a.a.a.200 is directly connected, Vlan1501
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.251.0 is directly connected, Vlan251
S*   0.0.0.0/0 [1/0] via a.a.a.201

[Georg Pauwen]

Hello,

I am lost on your physical setup. The output is from at1-wanstaack-1, while your 3750, the one with the two default static routes, is datacenter1-wanstack-1 ? Can you make a schematic drawing of your physical setup ? The idea with showing the routing table was to find out why NAT traffic gets routed even without the default routes, but IPSec traffic doesn't. A TRACEROUTE for both a NAT and IPSec destination would work, too...

[Georg Pauwen]

John,

first, I would fix the BGP load balancing and redundancy. There are several ways to configure this. The links below gives sample configurations for dual BGP with community string load balancing and dual BGP with AS-prepend load balancing.

http://showipbgp.com/bgp-configurations/40-cisco/118-4-4-2-cisco-dual-bgp-with-community-string-load-balancing.html

http://showipbgp.com/bgp-configurations/40-cisco/114-4-3-2-cisco-dual-bgp-with-as-prepend-load-balancing.html

If that gets too confusing, post the config of the 2900 router (the one connected to both ISPs) and we can put the right config in there.

Either way, with both configs you need two default routes pointing to both ISPs respectively. The keyword 'secondary' would not be necessary. I am not sure how this would influence your IPSec traffic. Can you post the configs of the 3750 stack and the 3845 as well ?

[John Gentle]

Here are our three configs from the switch.  Thank again.  I have been stumped off and on since December of last year and now it is sort of a problem that the more I think about, the more I need to resolve before it bites me hard.

3750 Stack Configuration

datacenter1-wanstack-1#sh run
Building configuration...

Current configuration : 7431 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
service internal
service unsupported-transceiver
!
hostname datacenter1-wanstack-1
!
boot-start-marker
boot-end-marker
!
no logging console
!
aaa new-model
!
aaa session-id common
clock timezone EASTERN -4
switch 1 provision ws-c3750v2-24ts
switch 2 provision ws-c3750v2-24ts
system mtu routing 1500
vtp domain domain.com
vtp mode transparent
ip routing
ip name-server 172.16.250.12
!
no errdisable detect cause gbic-invalid
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause channel-misconfig (STP)
errdisable recovery cause pagp-flap
errdisable recovery cause dtp-flap
errdisable recovery cause link-flap
errdisable recovery cause sfp-config-mismatch
errdisable recovery cause gbic-invalid
errdisable recovery cause l2ptguard
errdisable recovery cause psecure-violation
errdisable recovery cause port-mode-failure
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause pppoe-ia-rate-limit
errdisable recovery cause mac-limit
errdisable recovery cause vmps
errdisable recovery cause storm-control
errdisable recovery cause inline-power
errdisable recovery cause arp-inspection
errdisable recovery cause loopback
errdisable recovery cause small-frame
errdisable recovery interval 360
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 251
!
vlan 500
  private-vlan primary
  private-vlan association 2000-2002
!
vlan 1501-1502
!
vlan 2000
 name To-3845
  private-vlan community
!
vlan 2001
  private-vlan community
!
vlan 2002
  private-vlan community
!
interface FastEthernet1/0/1
 description datacenter1-edge-1 Ge0/1
 switchport private-vlan mapping 500 2000-2002
 switchport mode private-vlan promiscuous
!
interface FastEthernet1/0/2
 description datacenter1-core-1 Ge0/0
 switchport private-vlan host-association 500 2000
 switchport mode private-vlan host
!
interface FastEthernet1/0/3
 switchport access vlan 1501
!
interface FastEthernet1/0/4
!
interface FastEthernet1/0/5
!
interface FastEthernet1/0/6
!
interface FastEthernet1/0/7
!
interface FastEthernet1/0/8
!
interface FastEthernet1/0/9
!
interface FastEthernet1/0/10
!
interface FastEthernet1/0/11
!
interface FastEthernet1/0/12
!
interface FastEthernet1/0/13
!
interface FastEthernet1/0/14
!
interface FastEthernet1/0/15
!
interface FastEthernet1/0/16
!
interface FastEthernet1/0/17
!
interface FastEthernet1/0/18
!
interface FastEthernet1/0/19
!
interface FastEthernet1/0/20
!
interface FastEthernet1/0/21
!
interface FastEthernet1/0/22
 description To datacenter1-manswitch-1
 switchport access vlan 251
!
interface FastEthernet1/0/23
 description ISP1 100mb
 switchport access vlan 1501
 speed 100
 duplex full
!
interface FastEthernet1/0/24
 description datacenter1-edge-1 Ge0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface FastEthernet2/0/1
 description datacenter1-edge-2 Ge0/1
 switchport private-vlan mapping 500 2000-2002
 switchport mode private-vlan promiscuous
 shutdown
!
interface FastEthernet2/0/2
 description datacenter1-core-2 Ge0/0
 switchport private-vlan host-association 500 2000
 switchport mode private-vlan host
 shutdown
!
interface FastEthernet2/0/3
 description RMSG Switch Gi1/0/1
 switchport private-vlan host-association 500 2000
 switchport mode private-vlan host
!
interface FastEthernet2/0/4
!
interface FastEthernet2/0/5
!
interface FastEthernet2/0/6
!
interface FastEthernet2/0/7
!
interface FastEthernet2/0/8
!
interface FastEthernet2/0/9
!
interface FastEthernet2/0/10
!
interface FastEthernet2/0/11
!
interface FastEthernet2/0/12
!
interface FastEthernet2/0/13
!
interface FastEthernet2/0/14
!
interface FastEthernet2/0/15
!
interface FastEthernet2/0/16
!
interface FastEthernet2/0/17
!
interface FastEthernet2/0/18
!
interface FastEthernet2/0/19
!
interface FastEthernet2/0/20
!
interface FastEthernet2/0/21
 switchport access vlan 251
!
interface FastEthernet2/0/22
!
interface FastEthernet2/0/23
 description ISP2 100mb
 switchport access vlan 1502
 speed 100
 duplex full
!
interface FastEthernet2/0/24
 description datacenter1-edge-2 Ge0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface Vlan1
 no ip address
!
interface Vlan251
 ip address 172.16.251.98 255.255.255.0
!
interface Vlan500
 ip address x.x.x.7 255.255.254.0
 private-vlan mapping 2000-2002
!
interface Vlan1501
 ip address a.a.a.205 255.255.255.248
!
interface Vlan1502
 ip address y.y.y.141 255.255.255.248
!
ip classless
ip route 0.0.0.0 0.0.0.0 a.a.a.201
ip route 0.0.0.0 0.0.0.0 y.y.y.137 2
ip http server
ip http secure-server
!
snmp-server community nagios-ro RO
!
!
datacenter1-wanstack-1#


2900 Router Providing BGP

datacenter1-edge1-1#sh run
Building configuration...

Current configuration : 4280 bytes
!
! Last configuration change at 13:15:55 EASTERN Wed Aug 24 2016 by myadmin
version 15.3
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
no service password-recovery
no service dhcp
!
hostname datacenter1-edge1-1
!
boot-start-marker
boot-end-marker
!
!
security authentication failure rate 3 log
security passwords min-length 6
logging buffered 50000
logging rate-limit 10000
logging console warnings
no logging monitor
!
aaa new-model
!
!
aaa session-id common
clock timezone EASTERN -4 0
!
no ip bootp server
ip domain name domain.com
ip name-server 172.16.250.12
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
license udi pid CISCO2911/K9 sn
!
!
!
redundancy
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description To datacenter1-wantsack1 Fa1/0/24
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 duplex auto
 speed auto
 no mop enabled
!
interface GigabitEthernet0/0.1501
 description ISP1 100mb
 encapsulation dot1Q 1501
 ip address a.a.a.202 255.255.255.248
 ip access-group Outbound out
 ip virtual-reassembly in
!
interface GigabitEthernet0/0.1502
 description ISP2 100mb
 encapsulation dot1Q 1502
 ip address y.y.y.138 255.255.255.248
 ip access-group Outbound out
 ip virtual-reassembly in
!
interface GigabitEthernet0/1
 description datacenter1-wanstack Fa1/0/1
 ip address x.x.x.2 255.255.254.0
 no ip redirects
 no ip unreachables
 ip local-proxy-arp
 ip route-cache same-interface
 glbp 0 ip x.x.x.1
 glbp 0 priority 110
 glbp 0 preempt delay minimum 30
 duplex auto
 speed auto
 no mop enabled
!
interface GigabitEthernet0/2
 ip address 172.16.251.97 255.255.255.0
 duplex auto
 speed auto
!
router bgp MYASNUMBER
 bgp router-id x.x.x.1
 bgp log-neighbor-changes
 network x.x.x.0 mask 255.255.254.0
 neighbor a.a.a.201 remote-as 1111
 neighbor a.a.a.201 route-map localonly out
 neighbor y.y.y.137 remote-as 2222
 neighbor y.y.y.137 route-map localonly out
!
ip forward-protocol nd
!
ip as-path access-list 10 permit ^$
no ip http server
ip http authentication local
no ip http secure-server
!
ip route a.a.a.201 255.255.255.255 GigabitEthernet0/0.1501
ip route y.y.y.137 255.255.255.255 GigabitEthernet0/0.1502
!
ip access-list extended ACL_SSHACCESS
 permit ip x.x.x.0 0.0.0.31 any
ip access-list extended Outbound
 deny   ip 192.168.0.0 0.0.255.255 any log-input
 deny   ip 172.16.0.0 0.15.255.255 any log-input
 deny   ip 10.0.0.0 0.255.255.255 any log-input
 deny   udp any any eq netbios-ns
 deny   udp any any eq netbios-dgm
 deny   udp any any eq netbios-ss
 permit ip x.x.x.0 0.0.1.255 any reflect alliptraffic timeout 120
!
route-map localonly permit 10
 match as-path 10
!

datacenter1-edge1-1#

3845 Router doing IPSEC and NAT in our datacenter

datacenter1-core1-1#sh run
Building configuration...


Current configuration : 66924 bytes
!
! Last configuration change at 02:32:50 UTC Wed Sep 28 2016 by myadmin
! NVRAM config last updated at 23:01:43 UTC Mon Oct 3 2016 by myadmin
! NVRAM config last updated at 23:01:43 UTC Mon Oct 3 2016 by myadmin
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname datacenter1-core1-1
!
boot-start-marker
boot-end-marker
!
aaa new-model
!
aaa session-id common
!
crypto pki token default removal timeout 0
!
!
dot11 syslog
ip source-route
!
ip cef
!
!
!
!
ip name-server 172.16.201.12
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
voice-card 0
!
license udi pid CISCO3845-MB sn
!
redundancy
!
track 1 interface GigabitEthernet0/0 line-protocol
!
track 2 interface GigabitEthernet0/1 line-protocol
!
interface GigabitEthernet0/0
 description datacenter1-wanstack1.1 Fe1/0/4
 ip address x.x.x.5 255.255.254.0
 ip access-group ACL_PRIMARY-WAN-FIREWALL in
 ip nat outside
 ip virtual-reassembly in
 standby 2 ip x.x.x.4
 standby 2 timers 1 3
 standby 2 priority 200
 standby 2 preempt
 standby 2 track 1 decrement 110
 standby 2 track 2 decrement 110
 duplex auto
 speed auto
 media-type rj45
 crypto map AES_MAP
!
interface GigabitEthernet0/1
 no ip address
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/1.100
 encapsulation dot1Q 100
 ip address 172.16.100.252 255.255.254.0
 ip access-group ACL_PN in
 ip nat inside
 ip virtual-reassembly in
 standby 1 ip 172.16.100.254
 standby 1 timers 1 3
 standby 1 priority 200
 standby 1 preempt
 standby 1 track 1 decrement 110
 standby 1 track 2 decrement 110
!
interface GigabitEthernet0/1.105
 encapsulation dot1Q 105
 ip address 172.16.105.252 255.255.255.0
 ip access-group ACL_SFP in
 ip nat inside
 ip virtual-reassembly in
 standby 1 ip 172.16.105.254
 standby 1 timers 1 3
 standby 1 priority 200
 standby 1 preempt
 standby 1 track 1 decrement 110
 standby 1 track 2 decrement 110
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 x.x.x.1

ip route 192.168.1.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.2.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.3.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.5.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.6.0 255.255.255.0 GigabitEthernet0/0
!
snmp-server community nagios-ro RO
!
control-plane
!
mgcp profile default
!
line con 0
 logging synchronous
line aux 0
line vty 0 4
 access-class ACL_SSHACCESS in
 transport input ssh
!
scheduler allocate 20000 1000
ntp server 172.16.201.16
end

datacenter1-core1-1#

[Georg Pauwen]

John,

I might be missing something, but since you say the 3845 is doing NAT and IPSec, I cannot find either the ip nat inside or outside source statements, nor the crypto map that is mentioned, AES_MAP. Is this the full config of the 3845 ?

[John Gentle]

I took a bit out and should have left some of those as examples.  Sorry and again, much thanks!

3845 Router doing IPSEC in our datacenter

datacenter1-core1-1#sh run
Building configuration...


Current configuration : 66924 bytes
!
! Last configuration change at 02:32:50 UTC Wed Sep 28 2016 by myadmin
! NVRAM config last updated at 23:01:43 UTC Mon Oct 3 2016 by myadmin
! NVRAM config last updated at 23:01:43 UTC Mon Oct 3 2016 by myadmin
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname datacenter1-core1-1
!
boot-start-marker
boot-end-marker
!
aaa new-model
!
aaa session-id common
!
crypto pki token default removal timeout 0
!
!
dot11 syslog
ip source-route
!
ip cef
!
!
!
!
ip name-server 172.16.201.12
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
voice-card 0
!
license udi pid CISCO3845-MB sn
!
redundancy
!
track 1 interface GigabitEthernet0/0 line-protocol
!
track 2 interface GigabitEthernet0/1 line-protocol
!

crypto isakmp policy 10
 encr aes 256
 authentication pre-share
crypto isakmp key datacenter1-PN1.9UwAGVwsfsfahtryu63545k21ddfgjbBik address c.c.c.2
crypto isakmp key datacenter1-PN2.sOH52g4Ok5TBvJCAFQtcYwrOjC0OPKxpp1 address d.d.d.98
crypto ipsec transform-set AES_SET esp-aes 256 esp-sha-hmac
!
crypto map AES_MAP 10 ipsec-isakmp
 set peer c.c.c.2
 set transform-set AES_SET
 match address ACL_PN1
crypto map AES_MAP 12 ipsec-isakmp
 set peer d.d.d.98
 set transform-set AES_SET
 match address ACL_PN2

!
interface GigabitEthernet0/0
 description datacenter1-wanstack1.1 Fe1/0/4
 ip address x.x.x.5 255.255.254.0
 ip access-group ACL_PRIMARY-WAN-FIREWALL in
 ip nat outside
 ip virtual-reassembly in
 standby 2 ip x.x.x.4
 standby 2 timers 1 3
 standby 2 priority 200
 standby 2 preempt
 standby 2 track 1 decrement 110
 standby 2 track 2 decrement 110
 duplex auto
 speed auto
 media-type rj45
 crypto map AES_MAP
!
interface GigabitEthernet0/1
 no ip address
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/1.100
 encapsulation dot1Q 100
 ip address 172.16.100.252 255.255.254.0
 ip access-group ACL_PN in
 ip nat inside
 ip virtual-reassembly in
 standby 1 ip 172.16.100.254
 standby 1 timers 1 3
 standby 1 priority 200
 standby 1 preempt
 standby 1 track 1 decrement 110
 standby 1 track 2 decrement 110
!
interface GigabitEthernet0/1.105
 encapsulation dot1Q 105
 ip address 172.16.105.252 255.255.255.0
 ip access-group ACL_SFP in
 ip nat inside
 ip virtual-reassembly in
 standby 1 ip 172.16.105.254
 standby 1 timers 1 3
 standby 1 priority 200
 standby 1 preempt
 standby 1 track 1 decrement 110
 standby 1 track 2 decrement 110
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
ip nat inside source static tcp 172.16.100.41 25 x.x.x.50 25 extendable
ip nat inside source static tcp 172.16.100.41 443 x.x.x.50 443 extendable
ip nat inside source static tcp 172.16.100.12 636 x.x.x.51 636 extendable
ip nat inside source static udp 172.16.100.12 636 x.x.x.51 636 extendable
ip nat inside source static tcp 172.16.100.138 443 x.x.x.52 443 extendable
ip nat inside source static tcp 172.16.100.138 943 x.x.x.52 943 extendable
ip route 0.0.0.0 0.0.0.0 x.x.x.1

ip route 192.168.1.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.2.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.3.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.5.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.6.0 255.255.255.0 GigabitEthernet0/0
!
ip access-list extended ACL_NAT
 remark *** Start of Denys for PN1
 deny   ip 172.16.100.0 0.0.1.255 192.168.6.0 0.0.0.255
 remark *** Start of Denys for PN2
 deny   ip 172.16.100.0 0.0.1.255 192.168.5.0 0.0.0.255
 remark *** Permit Everything not denied above
 permit ip any any
ip access-list extended ACL_PN2
 permit ip 172.16.100.0 0.0.1.255 192.168.5.0 0.0.0.255
ip access-list extended ACL_PN1
 permit ip 172.16.100.0 0.0.1.255 192.168.6.0 0.0.0.255
 
 snmp-server community nagios-ro RO
!
control-plane
!
mgcp profile default
!
line con 0
 logging synchronous
line aux 0
line vty 0 4
 access-class ACL_SSHACCESS in
 transport input ssh
!
scheduler allocate 20000 1000
ntp server 172.16.201.16
end

datacenter1-core1-1#