Network design query

[darren-carr]

Hi All,

I have a query that I hope you can help me with? I am in the process of planning our new network. Our business is changing from hosting its own data centre, to moving it to a professional facility. We have 120 users, over 100 servers (physical and virtual) and three sites (main premise, data centre, dr site). The new network will connect all three.

Our new WAN links are almost ordered. We will be making use of a managed MPLS IP VPN, with a 100M access rate at each site.

I am currently focusing on the desing of the network at the main business premise. We have a significant investment in Cisco 2960 & 3750 switches and Fortinet firewall appliances. I plan to re-use these in the design.

Our current LAN is very flat and I want to segment the network. My plan is to create a number of VLANs, enable the Inter VLAN routing on the 3750 and then attach the 3750 to the Fortinet appliance which will provide stateful firewalling and traffic policin based on the VLAN (subnet) addresses. It is important that the traffic be routed as quickly as possible from this site to our prod and dr data centres.

The 2960's act as the access layer, the 3750 as the distribution layer. The 2960's will connect via port channels (layer 2) to the 3750's and the VLAN interfaces will be configured on the 3750.

I was then planning on creating a VLAN on the 3750 to connect to the Fortigate appliance with a /29 address to limit the addresses used whilst also providing some flexibility for any future design changes.

I want to implement a little security between the VLANs on the 3750 switches. I have a question about this coming up.

I then plan to use the Fortigate appliance to do basic traffic policing based on source/destination addresses.

The WAN routers will connect to the Fortinet appliance on a Gigabit copper interface. The WAN routers will run HSRP between themselves and only one router will be active at any one time.

The failover will be managed by the Fortigate and Cisco routers.

I plan to define those addresses hosted at the other data centres and associate them with the interface associated with the WAN.

I will then define the routing on the firewall for the two other data centres through summary routes for each of the sites. We will run static routing from the Cisco 3750 to the Fortigate and Fortigate to WAN router. We have no other networks/sites and won't have any others in the future.

Does this design sound reasonable? I am looking for some feedback. I can provide a drawing tomorrow if this would prove to be useful.

Thanks,

Darren

[Jon Marshall]

Darren

I plan to define those addresses hosted at the other data centres and associate them with the interface associated with the WAN.

What do you mean by the above ?

I will then define the routing on the firewall for the two other data centres through summary routes for each of the sites. We will run static routing from the Cisco 3750 to the Fortigate and Fortigate to WAN router. We have no other networks/sites and won't have any others in the future.

I am a little unclear on the WAN connectivity. Are you going to use one MPLS circuit connected to the HSRP active for both DC and DR and then use the other router purely as failover ? If so in your earlier you said -

It is important that the traffic be routed as quickly as possible from this site to our prod and dr data centres.

would it not make sense to simply have both links active at the same time ?

Jon

[darren-carr]

Hi Jon,

Thanks for the prompt reply. You really are dedicated to helping people. I appreciate your help.

With respect to your first query. I plan to make use of 10.1.x.x at my business premise and 10.2.x.x at the prod data centre, and lastly 10.3.x.x at the DR site. I have defined subnets on 10.1.x.x for general users, IT admin, IT developers, etc. At the prod data centre I have defined similar subnets for 10.2.x.x for prod server, etc.

The plan for the WAN was to have 2 x Cisco 3845's configured using HSRP (one active, one standby). A interface off the router would then be patched directly into the Fortigate appliance (Gig interface). Fortigate is capable of detecting a L1/L2 failure upstream and can failover to the other circuit if required.

With this in mind I was thinking of creating policies from the inside interface to the interface that the WAN is patched into i.e. port(inside) source = 10.1.x.x port(wan) destination = 10.2.1.x and then routing it over the WAN.

The second link is purely for redundancy as it is costly to operate two at once.

Each site has two physical lead ins from the street that provides complete physical redundancy. We have two connections at each site into a managed MPLS IP VPN cloud.

Does this make sense? If not I can upload a drawing tomorrow that may provide better understanding.

Apologies if this is a bit vague or sounds like I am being lazy. I've been working on this all day. I'm in Australia and about to sign off for the evening!

I appreciate any feedback/input you may have.

Cheers,

Darren

[Jon Marshall]

Darren

Most of what you have proposed sounds fine.

I am still not fully understanding the WAN routers to fortinet connectivity. When you say an interface off the WAN router is patched directly into the fortigate does that apply to both routers. A common setup is to have a L2 switch between the WAN routers and the firewall. Then the outside interface of the firewall and the 2 LAN interfaces on the routers share a common subnet and so you can use HSRP on the routers.

If you patch the active router directly to the fortigate how does HSRP come into it ? Is the redundancy going to be a manual thing ie. you need to patch the other router in if it fails.

HSRP would surely on work with a common L2 subnet but if the WAN routers are patched directly to the fortigate then unless the fortigate is acting as a L2 switch then how do HSRP messages go between the LAN interfaces on the routers.

So this is the only bit i am not really clear on ie. -

1) are you patching both routers directly to the fortigate on different fortigate interfaces ? - if HSRP doesn't come into it

2) are you only patching one WAN router ? If so this would then require manual switchover and still HSRP doesn't come into it

Jon

[darren-carr]

Hi Jon,

Sorry for the confusion.

I hope this answers you question.

The two routers at each site will be managed by the carrier. It is my understanding that the routers will be connected together by the carrier and this connection is what they will use to run HSRP between the devices.

I am a little concerned about how the Fortigate will detect the link down between itself and the Cisco router. I will need to read up again on HSRP.

I've attached an image regarding what I am trying to achieve. Hope this explains it a little better?

I've attached a

[Jon Marshall]

Darren

Hmm, it depends on how the fortigate reacts if the HSRP active gateway moves to the other router.

There is a an advantage to directly connecting the WAN routers to the fortigates.

That advantage is that because there is no switch between the firewalls and the WAN routers then the fortigate will be able to tell immediately if the interface on the active router has gone down. If there was a switch then you would need some sort of IP SLA functionality on the fortigate to ping the WAN router because obviously the LAN interface of the active router could go down but the fortigate interface is still up because it is connected to the switch.

I'm assuming the ISP will be doing interface tracking with HSRP so if the WAN interface on the active router goes down then the HSRP priority will be reduced and the other WAN router preempts.

So i don't think you need to read up on HSRP. What you do need to work out is how the fortigates react to HSRP switchover. With a switch in between the routers and firewalls the active firewall would simply continue to forward packets via L2 to the new HSRP active router. But there is no direct path from the active fortigate to the new HSRP active router in your design. So i'm wondering how the fortigates react to this.

1) if the LAN interface of the active WAN router fails then that should not be a problem ie. because the foritgate is directly connected then it's interface will also go down so the fortigate and the WAN router should both failover to the standby devices and everything should work as expected.

2) If the WAN interface fails and the HSRP priority is decreased so the other WAN router becomes active how does the fortigate handle this ie. the active firewall only has an indirect connection to the new active WAN router via the firewall interconnect link - can it use this link to get to the WAN router. Even if it can use this link the standby firewall is still in standby mode so would it forward traffic to the WAN router ie. in active/standby usually only the active firewall can forward traffic.

3) It may be that if the WAN interface fails on the HSRP active router then the HSRP priority is not decreased ie. they leave this router as the as the HSRP active and simply route the traffic received from the fortigate active firewall across the WAN router interconnect. This would work.

So you need to understand how the carrier is going to configure these routers and also if they go with 2) how your fortigates will react. I don't think 2) will work, or at least it wouldn't with ASAs because the standby firewall does not pass traffic and there is nothing to tell the active firewall to failover because it's outside interface is still up ie. the LAN interface of the WAN router isn't down, it;s just that the HSRP priority has been reduced so the other router becomes active.

The other problem is return traffic. If return traffic arrives at non-active router, again how does it get to the active fortigate assuming the active fortigate is connected to the HSRP active WAN router. Although again the ISP might be ensuring return traffic always comes down the link it went out on.

Although i said it was an advantage it does seem to complicate issues. This is why a standard setup for this sort of thing is to have a L2 switch or 2 L2 switches interconnected in your case, between the firewalls and routers. Then the active firewall has a path to both routers. With this setup though you do need to be able to check from the firewalls the status of the router LAN interfaces and i don't know if the fortigates have that functionality.

Jon

[darren-carr]

Hi Jon,

Thanks again for the response.

You are correct in assuming that the ISP will be doing the interface tracking with HSRP, so if there was a failure on the Active router upstream then the router would failover over to the standby device. The reason I wanted to read up on HSRP was that I am unclear as to what state the router could then place the interface facing the Fortigate into i.e. could it change the interface status to down? The Fortigate can monitor the physical attributes of its connection to the router.

I will spend some time with the carrier to discuss the desing.

Thanks again for your help. I'll let you know how I go.

Cheers mate

Darren

[Jon Marshall]

Darren

Just for your info, with HSRP tracking the LAN interface would not be shutdown only it's priority decreased enough that the standby router will preempt.

So you definitely need to talk to the carrier about this because i can't see how the active fortigate will get to the new active WAN router ie. the former standby router.

Jon

[campbell]

Hi Darren

I am not sure that the fortigate will give you the fail over you are hoping to achieve. I would assume the provider is giving you redundant IP paths rather than redundant interfaces. That is, HSRP will be presented to you so that you can use the HSRP IP address as your gateway out of your network. If you connect the fortigate in the way, I understand you, then the HSRP will not work and you will end up to two active HSRP nodes.

The fortigate failover mechanism is designed more for two different Internet provider connections rather than receiving one redundant connection. Also, why would you want to limit yourself by only ever utilising one 100Mb connection, when you could design your network to utilise both with a decent routing policy on both ends.

Also, I do not understand why you would separate your VLANS by a layer3 interface on the switches rather than using the fortigate firewalls. By using the switches, you will have to define ACL's per interface and then also configure the firewalls too, whereas, if you are using separation by firewall, then there is only one place for a security policy, although your suggested way would be better if security is not a concern and were hoping to achieve better throughput. I would ask then why then you would split your hosts into vlans, if the latter were true, as there is no real advantage to putting hosts into separate vlans unless you wanted to provide inter-vlan security, in which case, I would suggest you do this with the firewall.

Regards,

Ed

Security Consultant

Sent from Cisco Technical Support iPad App

[Jon Marshall]

Darren

Ed said -

That is, HSRP will be presented to you so that you can use the HSRP IP address as your gateway out of your network. If you connect the fortigate in the way, I understand you, then the HSRP will not work and you will end up to two active HSRP nodes.

Apologies but I missed this. I assumed from your diagram that there was an interconnect between the routers. But even if there was this interconnect cannot carry the HSRP hello's for different interfaces on the routers. So Ed is spot on when he says both routers would be HSRP active.

So if the carrier wants to run HSRP you need switches between the routers and firewalls.

Ed - thanks for catching that, i completely missed it which for a LAN switching person is kind of embarrassing

Jon

[darren-carr]

Hi Ed,

Thanks for your input.

You are correct with your first paragraph and I share the same concerns as the Fortigate would see its connection to the router as up even with an upstream failure.

We do use the Fortigate device as you have suggested in paragraph two for our internet connections.

With respect to your query in the secon paragraph it is more about cost. Our standby link is quoted as half the price of the active link.

I now plan to use the Fortigate to do the inter-vlan security/routing. It makes more sense to do it this way. I was playing with the design yesterday and I must admit I wasn't thinking too clearly. It certainly makes more sense and is easier to configure this was as opposed to maintaining ACL's on the layer 3 switch.

I guess for the routing the only option I have is to  drop the two router 'inside' facing interfaces into a layer 2 switch, in the same vlan as the Fortigate interface and route to both routers (active/active). I could then split the 100M in two (50M/50M) or double the bandwidth to 100M/100M?

Does this seem reasonable?

Thanks again for your input.

Darren

[Mohamed Sobair]

Hello Daren,

I personally dont recommend/suggest this kind of setup as you mentioned. I dont see a reason of why you make the fortigate perform the Layer-3 routing while you have a capable of Layer-3 Switches that can perform it.

The reason I am saying this is because , you have a switches that perform layer-3 switching even more rapidly than the firewall does because the firewall is primarily not designed to perform this functionality. Its even make more sense if you want to route/filter or apply QoS in the future to have all Layer-3 functionality performed by the 3750 Stack.

It wont increase the config for you, All what you want is to have layer-3 interfaces on the switches for the inter vlan routing and another layer-3 point to point interfaces for the connectivty with the fortigates.

You can stick with the current Setup of having Active/Passive Scenario on the fortigates and One upstream provider active at a time with proper Implementation which should include IP Sla.

You should have 2 default routes from the Stack of 3750 with IP Sla tracking , the active one points to the Primary fortigate while the Backup one points to the Standby fortigate.

With respect to your Internet routers, you should have the 2960 in place as you have drawn in the diagram, and have HSRP Active/Standby between them.

Each of the Internet routers should have 1 default route along with Tracking option , to track its WAN connection and loses its HSRP Active Gateway functionality once it loses its WAN connection and since this is the provider Job , you dont have to worry about it , you can just inform them about this type of confi/implemetation. and I am sure this is what they would perform in the end.

The FOrtigate Firewalls doesnt need 2 default routes, both fortigates should have 1 default route pointing to the HSRP VIP address of the routers and it should point them to the correct Active router based on the previous implementation of the WAN routers Connection and WAN Tracking.

I beleive this is the optimal design for you, and this type of Scenario is more applicable for most similar Setup according to your requirement.

HTH

Mohamed

[darren-carr]

Hi Mohamed,

Thanks too for your input. I appreciate what you have outlined above. I do think that doing the inter-vlan routing through the Fortigate will be easier than maintaining ACL's for the VLANs. The appliance I have is an enterprise class device that is currently doing next to nothing with respect to performance. Managing this through a GUI certainly makes life a lot easier. Happy to be challenged on this though.

With respect to the HSRP operation I am fortunate in that I have a couple of switches in place with a bit of spare capacity to patch the the Cisco routers (inside) interface and Fortigate interface into. The carrier has confirmed this as their preferred solution.

I can easily change to a ACTIVE/ACTIVE solution if performance becomes a problem. The budget and design allows for this. I am just conscious of the costs associated.

Thanks again for all of your input, it has been of great help to me.

Cheers,

Darren

[campbell]

Hi Darren,

I am not sure whether Mohammed understands how the fortigate firewalls' redundancy works, so just to clarify. Fortigate firewalls support an active/passive state, with no traffic being put through the passive firewall at all. Mohammed suggested two default routes from your switches, one to the active and one to the standby. This configuration is not support by the fortigate in active/passive state. You could go with an active/active scenario, but then you would not need two defaults route either, as a multicast mac address is used for the cluster ip address. Also a point to note, some applications like https cannot be load balanced across a fortigate cluster.

To be clear about my inter-vlan separation argument. If security is paramount, and it may not be, then separation by the firewall is the way to go. Switches will not give you stateful checking of packets at all and will be more easy to bypass.

If performance is what you need, then as I stated before, perform separation on the switches then apply access-lists to each interface. However, in my experience and as networks and policies grow, this can be laborious and often forgotten about.

However, if you would tell me the fortigate platforms you are using, then I would be able to judge better whether they will interfere with performance. The fotigate firewall uses ASIC based chips to achieve wire speed firewalling. This is the case with all of their firewalls, so I do not see that performance would be an issue. They also support QoS and bandwidth throttling to the application level. I would hardly think that they will be a problem considering the fact that it is highly unlikely you will have each and every server consuming bandwidth to each server in every other vlan. But as Mohammed suggests, and if security is not a concern, performance through the switches will be better, so you just have to decide on what the risk level is and choose the separation method to suit.

In your last paragraph of your reply to me, the active/active scenario. This method will give you one active, but clustered, IP address to use as you gateway out. So you would have to configure two separate routing interfaces to do what you are suggesting. There maybe a configuration option where it is possible to use an IGP to achieve load balancing over the two links but it depends on equipment at each end and your company's current skill level to support it.

So I would advise this, keep the design simple, make sure that there is the knowledge in your company to support it. If the design is simple then only a few easily identifiable things can go wrong. If it is complicated by various other details, then it will be complicated to support. Look at where you are now and then compare it to what you will have. Is there a need to implement a whole new design, does the business require it? Step back and see what the business requires before embarking on a further design effort. You may find that your needs are far more simplistic.

Good luck, with your design.

Regards,

Ed

Sent from Cisco Technical Support iPad App

[Mohamed Sobair]

Hello ED,

I just want to clarify one thing,

The Two default routes from the Stack Switch , is defined with one as primary (Active route) to the Active Fortigate, while the other is ONLY backup (NOT Active). The Backup ONLY Kicks in when the Primary route fails. this what I exactly meant.

Coming to the Security Point, Do you think letting the Fortigate [performs the (Inter-vlan routing) between Local user VLANs adds Security here, I personally see no difference, Since you would still need to add permission on the FW for that, just like when you have ACL on the Switch. its not only the performance is our constraint, but other Options as well.

The firewall would be a good advantage o protect from the Internet as well as from LAN to DMZ and Internet to DMZ , INternet to LAN. but I am not sure, for the LAN segments he needs to pass the traffic through the FW as well.

In Anyway, this is my Openion, and Have seen such similar setup is been done in several situation,

You could have different Openion though.

Thanks,

Mohamed