new firewall, vlan problems

[CiscoSteve3]

Hi guys,

I have a 3750X behind my new PA-3020.  All my vlans have IPs on the 3750 and therefore show as directly connected.  I want to move some vlans to the PA-3020.  I'm creating subinterfaces on my firewall, tagging them for the right vlans, etc.  But when I remove the SVI from my switch, the traffic is not getting there.  My research has told me to keep the vlan in the database, but not to give it an interface on the switch.  My default route is configured for the parent interface on the firewall, all traffic is getting there fine.  

Here is my test scenario:

Firewall ethernet 1/2 IP 10.38.0.1/29 is connected to 3750X int gig 1/0/1 ( a routed layer 3 interface with IP 10.38.0.5/29).

Firewall ethernet 1/2.100 IP 10.38.2.1/24, tagged 100

Workstation IP 10.38.2.6

3750x#

ip route 0.0.0.0 0.0.0.0 10.38.0.1 (default gateway)

ip route 10.38.2.0 255.255.255.0 10.38.2.1

I thought my workstation would be able to ping the FW eth1/2.100 - but it can't.  The switch can.

What am I doing wrong?  Is there a better/easier way to route some vlans through the switch and others through the firewall?

Thanks!

Steve

[jj27]

Hi Steve,

With the port in routed mode, there is no VLAN tag associated with the connection so you will not be able to tag other VLANs on the firewall and expect it to work.

In order to be able to tag VLANs across the link to your firewall, you would likely need to make the connection to Ethernet1/2 on the PA layer 2 on the switch.  You would make the switchport a trunk and allow the VLANs you would want to create a layer 3 interface for on the firewall.

The switchport configuration would look like something like this:

interface g1/0/1
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport trunk allowed vlan 100,xxx,xxx

Replace xxx with the other VLAN numbers.  

You would also likely need to create a new VLAN and a corresponding layer 3 interface on the switch for the 10.38.0.0/29 address space. Give the switch the .5 address, on the firewall Ethernet1/2.xxx (where xxx is that new VLAN) give it the .1 address.

This will require a maintenance window.  Make sense?

[CiscoSteve3]

Thanks for the advice, it does make more sense now.  I'll give this a go as soon as I can arrange a good time to do so.

Steve

[Jon Marshall]

Steve

If you are running subinterfaces on your firewall you cannot use a L3 routed port on the 3750.

It needs to be a trunk link so that traffic can get from the client to the subinterface on the firewall.

You are doing it the right way in terms of moving SVIs to the firewall for those vlans you want routed on the firewall.

So to make this work -

1) change the port on the 3750 to be a trunk link

2) create a new vlan just for the switch to firewall connection

3) create an SVI for this new vlan and give it the IP address originally on the routed port

then because you are using the main firewall interface and not a subinterface you need to make the native vlan on the trunk link the new vlan you created for the connection because those packets won't be tagged.

Then use the "switchport trunk allowed vlan <vlans>" command to allow that vlan and the other vlans you have subinterfaces for on the firewall.

Edit - only allow the new vlan you created and the vlans that have subinterfaces on the firewall on the trunk link. The other vlans should be routed off the switch.

Jon

[CiscoSteve3]

Thanks Jon, I was most of the way through this config but didn't do all of it correctly.  I will set this up when I can interrupt the traffic without causing too much trouble.  Appreciate your help!

[CiscoSteve3]

Update:

I tried the following in a very short maintenance windows last night:

3750x#

interface vlan 3

ip address 10.38.0.5 255.255.255.248

interface gigabit 1/0/1

description SW <> FW

switchport trunk encapsulation dot1q

switchport trunk native vlan 3

switchport trunk allowed vlan 3,100

switchport mode trunk

Now I'm not sure if my vlan 100 would have worked in this configuration as I did not have enough time to get that far (very small maintenance window).

I noticed right away that traffic from my other vlans (those routed on the switch) did not have internet access.  I assumed it could not get across the new trunk link since only vlans 3 and 100 were allowed, so I tried "switchport trunk allowed vlan all" but that didn't work and I'm assuming that is because there were not corresponding subinterfaces on the firewall.  I don't want to go there, I'd like to keep some vlans routed on the switch, but I need them to also have access to the internet via the firewall.

Am I missing something here?  Or can I not use both approaches?

Thanks guys!

Steve

[Jon Marshall · Accepted solution]

Steve

Traffic from the vlans routed on the switch should use vlan 3 to be routed across to the firewall.

I assume you left the routing you had in place and used the same IP on the SVI for vlan 3 on the switch so the switch and the firewall should have be able to route between each other.

There are a couple of things -

1) you may have to clear the arp cache on the router when you try this because the SVI will probably have a different mac address than the routed port you were using before.

2) the other potential problem is the native vlan and main interface on the firewall. Your firewall may or may not like this setup.

If it doesn't the way to get around it is to not use the main interface ie.

you create a new subinterface for vlan 3 on the firewall and then you change the native vlan on your switch to be something else, any unused vlan eg. vlan 999 so that vlan 3 packets are tagged.

If you do that you don't then need to allow the native vlan on the trunk link.

Either of the above could have been the problem and I would check the arp cache first,  but what you are trying to do is definitely possible.

Jon

[CiscoSteve3]

Great, thanks Jon.  I'll attempt it again in my next window, likely not until Friday night.

The routing was left in place, the IP on the SVI for vlan 3 was the same that had been on the interface g 1/0/1 previously, the default gateway was still in place for the next hop to the FW interface.  That route should have been ok.

The main interface on the FW is untagged, so that could be part of the problem, in which case a sub interface tagged 3 might be the solution.

When traffic in vlan 30 is forwarded to the default gateway, is the vlan tag dropped?  I guess I was thinking if traffic originates in vlan 30 then I would need to allow vlan 30 across the trunk.  No?

Also, the native vlan part confuses me.  Wouldn't you want the native vlans to match?  Why do I need it at all?  As I understand it, the native vlan will be used when untagged traffic reaches that link...but is there any untagged traffic?

So many questions, thanks for the help!

[CiscoSteve3]

I set up a completely identical environment with a spare switch I had on hand.  I used a different interface on the FW and changed the IPs from 10.x.x.x to 11.x.x.x, but otherwise identical.  I could not get it to work with any of the scenarios described.

I think the problem is that traffic tagged in vlans that do not exist on the FW is getting lost.  It worked perfectly for the vlan 100 with a subinterface tagged 100 on the FW.  But vlan 30 traffic routed on the switch doesn't work.

I'm not sure it's possible, maybe the firewall just doesn't like this setup?  Obviously I could put all my vlans on subinterfaces on the firewall and all routing would happen there.  This will work, but it creates more zones and policies to pay attention to, particularly for vlans where I don't need that level of security.  That said, if it works, I may need to just do it.

Do I lose any significant benefit by moving all routing to the firewall?  My layer 3 switch would become just a layer 2 switch...pros and cons?

[CiscoSteve3]

OK, I solved part of it.  The FW still needs a subinterface tagged with a vlan to match each vlan that is routed on the switch.  The IP of this subinterface is not in the routing table on the switch and is not used for any routing on the firewall, it just grabs the vlan traffic when it comes in to the main interface on the FW and forwards on to the default route in the firewall.  It seems like an odd setup since the gateway for these vlans is the SVI on the switch, but it works, so I'm happy.  Now I just need a good maintenance window to test this on my production equipment.

Thanks for the help so far!

[Jon Marshall]

Steve

The FW still needs a subinterface tagged with a vlan to match each vlan that is routed on the switch.

That doesn't make sense.

Any vlans that are routed on the switch are sent over vlan 3 so the firewall would never see tagged packets for any vlans that are routed on the switch.

Perhaps I have misunderstood and if it works then fine but you should only need subinterfaces on the firewall for the vlans you are routing off it and perhaps vlan 3 as discussed previously.

There is no reason why this wouldn't work, I have done s similar thing myself just not with your type of firewall.

Jon

[devils_advocate · Accepted solution]

The FW still needs a subinterface tagged with a vlan to match each vlan that is routed on the switch

I am also unsure what you mean by this as only Vlan 3 and Vlan 100 would need a 'tagged' subinterface because they are the only ones you have moved over to the PA appliance.

The Vlans which are still connected to the L3 switch would not tag their packets, they would simply use the default route up to the PA. 

We have a PA-200 in our test lab and it has subinterfaces configured on it.
The physical interface is untagged, doesn't have an IP address and is not part of a security zone.

The subinterfaces are all tagged with the relevant Vlan and are part of security zones.

See attachment for how mine is setup.

EDIT - Just a thought, if you look at your Virtual Router on the PA (the one assigned to the subinterfaces), are the subnets which are terminating on the Switch in there as static routes? I am just wondering if the PA appliance knows how to get back to the Switch for the Subnets which are a not directly connected to it. 

Thanks

[CiscoSteve3]

Good point about the static routes, for my test case I don't believe I added them.  I agree that it seems it should work without subinterfaces for vlans not routed on the FW, perhaps I've been missing yet another piece.

I'll have another go at my little lab setup today to see if I can't make it works the way it should.

Thanks!

[CiscoSteve3]

In the lab adding the static route for the vlan routed on the switch, not directly connected to the FW, did not change the situation.  

Let's review my config, maybe I'm missing something still (likely):

Switch#

int vl 3

ip add 10.38.0.5/29

int vl 30

ip add 10.38.1.5/24

int gi 1/0/1 (connected to FW int eth 1/3)

switchport trunk encapsulation dot1q

swithport trunk native vlan 3

swithcport mode trunk

switchport trunk all vlan 3,100

int gi 1/0/23

switchport mode access

switchport access vlan 100

int gi 1/0/24

switchport mode access

switchport access vlan 30

default gateway is 10.38.0.1

10.38.0.0/29 is directly connected via vlan 3

10.38.1.0/24 is directly connected via vlan 30

PA-3020#

int eth 1/3.3

ip add 10.38.0.1/29

tagged vlan 3

dhcp relay to 172.19.1.9, 172.19.1.11

int eth 1/3.100

ip add 10.38.2.1/24

tagged vlan 100

dhcp relay to 172.19.1.9, 172.19.1.11

Virtual router has route for 10.38.1.0/24 out int eth1/3.3 to next hop 10.38.1.5

I don't think it's a route problem though, because I'm not getting an address at all, so the traffic problem is occurring before it has an address to route, right?

When I plug a pc into switch int gi 1/0/23 on vlan 100, everything works as expected.  Get an address, get to the internet.

When I plug a pc into switch int gi 1/0/24 on vlan 30, I don't get an address.

Thinking it might just be dhcp issue, I did test with a static IP on the workstation, still no traffic gets through.

I don't know enough about how this works to back up this thought, but it feels like the vlan 30 traffic is getting to the firewall still tagged vlan 30 and the firewall drops it because it doesn't have a vlan 30.  When I add a subinterface in vlan 30, everything falls into place.  I know it should be tagged 3 since it was routed there over native vlan 3....

Hope the extra info and review here helps you guys help me think through this.  Thanks, I really appreciate the time and thought you're giving this.

Steve

[Jon Marshall]

Edit - ignore that, vlan 30 is presumably being routed on the switch ?

Jon