Cisco 6506 with 2 ISP connections

[WolfraiderNW]

I know this is a simple question but for some reason I cannot get it to work.

We have 2 ISP connections to our 6506 that we are using as a router. We also have a fwsm installed. ISP A has given us a public ip range of 10.10.10.0/23 and the second has given us several public IP ranges. We want the ability to route most of our NAT networks through ISP 2 with a few out ISP 1. We are not looking at ISP failover or anything at this time. We have tried creating a static route and route-maps.

All NATing works through ISP2 with the default gateway. How do we route all the 10.10.10.0 traffic to use ISP1 only. I have created a test vlan with the public ip address of 10.10.10.1 and while I can ping inside the network, I cannot ping outside. I am missing something really simple I know.

[fsebera]

Hi,

The 10.10.10.0/23 is not a public ip range, the 10.x.x.x/8 is private RFC1918 space and therefor your ISP must NAT if you need to reach a public destination on the Internet.

Do you receive the full Internet Routing Table from either of your ISPs? If you run the show ip route command, you should see 500,000+ prefixes if you are receiving the full table.

Are you running BGP with either of your ISPs?

Frank

[WolfraiderNW]

I used 10.10.10.0 on place of the real public ip addresses.

We did not recieve BGP from either ISPs. Mainly 1 ISP will not do BGP until we purchase an ID from ARIN. They recommend doing static routes only until we can purchase our own IP space and setup failover between the ISP's. We dont have the Routing Tables either.

[WolfraiderNW]

We have the traffic working correctly on the Cisco 6506 using a route-map and the 10.10.10.0/23 network. Now the isse is that the fwsm still doesnt work. We created a vlan that has the 10.10.10.0/23 network range in it and can successfully ping the outside world on the correct ISP. We added that vlan to the fwsm and it cannot ping anything from that vlan. Am I correct in assuming that traffic coming into the fwsm, while NATed should go out the default gateway of the fwsm (which is on a private network between the fwsm and 6506) and then the 6506 should route the traffic accordingly?

[Jon Marshall]

It's not clear exactly how you have the FWSM setup in relation to the 6500 ie. is the FWSM being used for both ISPs ?

If so is the logical setup -   internal vlans -> 6500 (MSFC) -> FWSM -> ISPs ?

In addition you mention using a route map for specific traffic so does the default route on the 6500 point to an FWSM inside interface for most traffic and then you use a route map to redirect some traffic to a different interface on the FWSM or something different ?

If you could provide more details on how the 6500 and FWSM are setup in relation to the ISPs in terms of interfaces etc. that would help.

Jon

[WolfraiderNW]

The fwsm is set to routed mode

The logical setup, from what I can tell, is

internal vlans -> 6500 -> fwsm (vlans are created on all devices)

fwsm does all NATing

fwsm -> 6500 -> ISPs

I checked and we have the fwsm set on private addresses for the outside vlan and it looks like we simply use the default route on the 6500 to route to ISP2, I created a route-map on the 6500 for ISP1 which works fine. The fwsm is used for both ISPs for NATing only.

[Jon Marshall]

What is confusing is that you say the internal vlans are routed on the 6500 but then you also suggest that 6500 used for routing on the outside of the FWSM.

This would mean traffic could be routed around the FWSM which I assume is not what is happening.

To help I am really going to need to understand the setup a lot better ie.

Do the FWSM only have one outside interface ?

If it does are both ISP routers in the same IP subnet

Are the internal vlan default gateways on the FWSM or the MSFC

For the outside interface(s) of the FWSM is there an SVI on the MSFC in the same IP subnet (it sounds like there is if you are applying PBR).

Jon

[WolfraiderNW]

I think this may help, here is our configs. This 6500 and fwsm was configured before I started working here and is a complete mess. I am in the process of cleaning it up and getting a standard config across all the different vlans.

6500

firewall multiple-vlan-interfaces
firewall module 1 vlan-group 1
firewall vlan-group 1  10,21-26,30-34,39,99,200,201,701-703,801,802,900-902
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
diagnostic bootup level minimal
access-list 10 permit 69.x.x.0 0.0.1.255
vlan internal allocation policy ascending
vlan access-log ratelimit 2000
vlan 701
 name Test-Office
!
vlan 702
 name nvctest
!
vlan 801
 name FWSM-Outside
!
vlan 802
 name Telcom
interface Null0
 no ip unreachables
!
interface TenGigabitEthernet2/1
 description MoTeleCom ISP
 ip address 69.x.x.229 255.255.255.254
 ip nat outside
 ip flow ingress
interface GigabitEthernet4/1
 description Internal VLANs
 switchport
 switchport trunk encapsulation dot1q
 switchport mode trunk
 ip dhcp snooping trust
interface GigabitEthernet4/3
 description Springnet ISP
 ip address 66.x.x.74 255.255.255.252
 ip nat outside
 ip flow ingress
interface Vlan701
 no ip address
!
interface Vlan702
 no ip address
!
interface Vlan801
 description FWSM Outside Network
 ip address 64.x.x.1 255.255.255.0
 ip policy route-map TelecomCommunity
!
interface Vlan802
 ip address 69.x.x.1 255.255.254.0
 ip policy route-map TelecomCommunity
ip nat translation timeout 1800
ip classless
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 66.x.x.73
ip route 64.x.x.0 255.255.255.0 64.x.x.2
ip route 76.x.x.0 255.255.252.0 76.x.x.2
route-map TelecomCommunity permit 10
 match ip address 10
 set ip next-hop x.x.148.228

fwsm

names
name 64.x.x.128 core
dns-guard
!
interface Vlan701
 nameif inside
 security-level 100
 ip address 192.168.101.1 255.255.255.0 
!
interface Vlan702
 nameif nvctest
 security-level 10
 ip address 192.168.102.1 255.255.255.0 
!
interface Vlan801
 nameif outside
 security-level 0
 ip address 64.x.x.2 255.255.255.0 
!
interface Vlan802
 nameif Telecom
 security-level 0
 ip address 69.x.x.2 255.255.254.0 
!
access-list optimization enable

access-list outside extended permit icmp any any 
access-list outside_access_in remark Permit ICMP Anywhere
access-list outside_access_in extended permit icmp any any 
access-list outside_access_in remark Allow Core Servers Unrestricted Access
access-list outside_access_in extended permit ip core 255.255.255.224 any 
access-list outside_access_in remark Port ranges that are never allowed to connect
access-list outside_access_in extended deny object-group Blocked-Services any any 
access-list outside_access_in remark Allow access to ports on all public pool IPs
access-list outside_access_in extended permit ip any object-group Public-IP-Ranges 
access-list inside_nat_static extended permit ip 192.168.101.0 255.255.255.0 core 255.255.255.224 
access-list inside_access_in extended permit ip any any 
access-list nvctest_nat_static extended permit ip 192.168.102.0 255.255.255.0 core 255.255.255.224 
access-list nvctest_access_in extended permit ip any any 
access-list Telecom_access_in extended permit ip any any 
access-list nvctest_nat_static2 extended permit ip 192.168.102.0 255.255.255.0 any 
no pager
logging enable
logging buffer-size 65535
logging buffered debugging
logging asdm informational
mtu nvctest 1500
mtu inside 1500
mtu outside 1500
mtu Telecom 1500
no failover
icmp permit any nvctest
icmp permit any inside
icmp permit any outside
icmp permit any Telecom
asdm history enable
arp timeout 14400
nat-control
global (outside) 110 76.x.x.1-76.x.x.253
global (outside) 1 interface
global (Telecom) 210 69.x.x.5-69.x.x.254 netmask 255.255.254.0
nat (nvctest) 210 192.168.102.0 255.255.255.0
nat (inside) 110 192.168.101.0 255.255.255.0
static (inside,outside) 192.168.101.0  access-list inside_nat_static norandomseq 
static (nvctest,Telecom) 192.168.102.0  access-list nvctest_nat_static2 
access-group nvctest_access_in in interface nvctest
access-group inside_access_in in interface inside
access-group outside_access_in in interface outside
access-group Telecom_access_in in interface Telecom
route outside 0.0.0.0 0.0.0.0 64.22.236.1 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:10:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 1:00:00 h225 1:00:00 mgcp 0:10:00
timeout mgcp-pat 0:10:00 sip 0:30:00 sip_media 0:10:00
timeout sip-invite 0:10:00 sip-disconnect 0:10:00
timeout pptp-gre 0:02:00
timeout uauth 0:10:00 absolute
dhcprelay enable nvctest
dhcprelay enable inside

[Jon Marshall]

Okay, thanks for that.

Firstly if those are our real public IPs can you obscure them eg. 62.x.x.2 so I can get the idea but the full IP isn't there.

So it looks like you have internal vlans eg. 701 and 702 that are routed on the FWSM.

What is still not clear is what SVIs you have on the 6500 for the outside and Telecom interfaces ?

Is there a separate SVI for each vlan eg. vlan 801 and vlan 802 ?

Can you post the SVI part of the 6500 (again editing the public IPs) and also the PBR configuration ?

I am assuming you do have SVIs for those vlans on the MSFC ?

Sorry to keep asking for more information but I need to see the whole picture.

Jon

[WolfraiderNW]

Oh, sorry about that, I thought I pasted all the config and missed half of it. I edited the post above with the full config and obscured

[Jon Marshall]

Okay, getting there :-)

When you obscure the IPs can you do first octet, then x.x and last octet as then I can work out what goes with what.

So your default route goes to SpringNet ISP.

Additional information, queries -

1) I need to see acl 10 used in the PBR.

2) on the 6500 on both L3 ports to the ISPs you have "ip nat outside" but there is no matching "ip nat inside" which there wouldn't be because the traffic is coming through the FWSM (unless there are other vlans you haven't shown on the 6500).

So I assume that NAT is actually done on the FWSM and those NAT statements on the L3 ports to the ISPs are not doing anything.

Is that correct ?

Jon

[WolfraiderNW]

access-list 10 permit 69.x.x.0 0.0.1.255

correct, all nat is coming through the fwsm.

[Jon Marshall]

So traffic from vlan 702 (nvctest) has the source IPs translated to 69.x.x.x as they go through the FWSM and then your PBR matches on the 69.x.x.x IPs and should send the traffic to the Telecom ISP.

Is that correct ?

If so now that I understand how it is all setup can we go back to the start and can you clarify exactly what isn't working ?

Jon

[WolfraiderNW]

Correct

If I ping from the 69.x.x.x vlan 802 from the 6500 I can get to the outside world just fine. If I ping from the 69.x.x.x vlan 802 from the fwsm I cannot see anything. It's as if the fwsm is not passing traffic through that vlan.

I also tried removing vlan 802 from the fwsm, changing the nat rules to ppint to outside instead of Telecom and changed the static 210 to outside. I figured that the fwsm would translate vlan 702 to 69.x.x.x and route the traffic through the default gateway which is pointed at an address on the 6500. PBR would then route the traffic through Telecom ISP

[Jon Marshall]

I am assuming in the following you are pinging an internet IP.

When you ping from the 69.x.x.x IP on the 6500 the 6500 simply uses it's default route and sends the traffic via SpringNet (note the PBR does not affect traffic sourced by the 6500 itself) although the return traffic comes back in via the Telecom ISP very probably.

But a ping from the FWSM is different because it is a firewall and not a router.

The default route on the FWSM points to the vlan 801 SVI IP on the 6500.

The FWSM has no route to use when the ping is sourced from the vlan 802 interface because that is in effect an outside interface so it would need a default route pointing to the vlan 802 SVI IP on the 6500.

But you can't have multiple default routes on the FWSM pointing out of different interfaces.

So any traffic from the nvctest vlan will always be routed via the outside interface and not the Telecom interface so the NAT won't happen as you want it to.

Your other approach of removing vlan 802 and using the outside interface for all communication should however have worked.

All traffic would be sent to the SVI for vlan 801 on the 6500 and then PBR should direct the traffic where you want it.

But you say that didn't work either ?

Jon