topic Network design query in Routing and SD-WAN

Network design query

darren-carr — Mon, 04 Mar 2019 21:22:23 GMT

Hi All,

I have a query that I hope you can help me with? I am in the process of planning our new network. Our business is changing from hosting its own data centre, to moving it to a professional facility. We have 120 users, over 100 servers (physical and virtual) and three sites (main premise, data centre, dr site). The new network will connect all three.

Our new WAN links are almost ordered. We will be making use of a managed MPLS IP VPN, with a 100M access rate at each site.

I am currently focusing on the desing of the network at the main business premise. We have a significant investment in Cisco 2960 & 3750 switches and Fortinet firewall appliances. I plan to re-use these in the design.

Our current LAN is very flat and I want to segment the network. My plan is to create a number of VLANs, enable the Inter VLAN routing on the 3750 and then attach the 3750 to the Fortinet appliance which will provide stateful firewalling and traffic policin based on the VLAN (subnet) addresses. It is important that the traffic be routed as quickly as possible from this site to our prod and dr data centres.

The 2960's act as the access layer, the 3750 as the distribution layer. The 2960's will connect via port channels (layer 2) to the 3750's and the VLAN interfaces will be configured on the 3750.

I was then planning on creating a VLAN on the 3750 to connect to the Fortigate appliance with a /29 address to limit the addresses used whilst also providing some flexibility for any future design changes.

I want to implement a little security between the VLANs on the 3750 switches. I have a question about this coming up.

I then plan to use the Fortigate appliance to do basic traffic policing based on source/destination addresses.

The WAN routers will connect to the Fortinet appliance on a Gigabit copper interface. The WAN routers will run HSRP between themselves and only one router will be active at any one time.

The failover will be managed by the Fortigate and Cisco routers.

I plan to define those addresses hosted at the other data centres and associate them with the interface associated with the WAN.

I will then define the routing on the firewall for the two other data centres through summary routes for each of the sites. We will run static routing from the Cisco 3750 to the Fortigate and Fortigate to WAN router. We have no other networks/sites and won't have any others in the future.

Does this design sound reasonable? I am looking for some feedback. I can provide a drawing tomorrow if this would prove to be useful.

Thanks,

Darren

Network design query

Jon Marshall — Tue, 23 Aug 2011 12:50:25 GMT

Darren

I plan to define those addresses hosted at the other data centres and associate them with the interface associated with the WAN.

What do you mean by the above ?

I am a little unclear on the WAN connectivity. Are you going to use one MPLS circuit connected to the HSRP active for both DC and DR and then use the other router purely as failover ? If so in your earlier you said -

It is important that the traffic be routed as quickly as possible from this site to our prod and dr data centres.

would it not make sense to simply have both links active at the same time ?

Jon

Network design query

darren-carr — Tue, 23 Aug 2011 13:19:07 GMT

Hi Jon,

Thanks for the prompt reply. You really are dedicated to helping people. I appreciate your help.

With respect to your first query. I plan to make use of 10.1.x.x at my business premise and 10.2.x.x at the prod data centre, and lastly 10.3.x.x at the DR site. I have defined subnets on 10.1.x.x for general users, IT admin, IT developers, etc. At the prod data centre I have defined similar subnets for 10.2.x.x for prod server, etc.

The plan for the WAN was to have 2 x Cisco 3845's configured using HSRP (one active, one standby). A interface off the router would then be patched directly into the Fortigate appliance (Gig interface). Fortigate is capable of detecting a L1/L2 failure upstream and can failover to the other circuit if required.

With this in mind I was thinking of creating policies from the inside interface to the interface that the WAN is patched into i.e. port(inside) source = 10.1.x.x port(wan) destination = 10.2.1.x and then routing it over the WAN.

The second link is purely for redundancy as it is costly to operate two at once.

Each site has two physical lead ins from the street that provides complete physical redundancy. We have two connections at each site into a managed MPLS IP VPN cloud.

Does this make sense? If not I can upload a drawing tomorrow that may provide better understanding.

Apologies if this is a bit vague or sounds like I am being lazy. I've been working on this all day. I'm in Australia and about to sign off for the evening!

I appreciate any feedback/input you may have.

Cheers,

Darren

Network design query

Jon Marshall — Tue, 23 Aug 2011 13:36:11 GMT

Darren

Most of what you have proposed sounds fine.

I am still not fully understanding the WAN routers to fortinet connectivity. When you say an interface off the WAN router is patched directly into the fortigate does that apply to both routers. A common setup is to have a L2 switch between the WAN routers and the firewall. Then the outside interface of the firewall and the 2 LAN interfaces on the routers share a common subnet and so you can use HSRP on the routers.

If you patch the active router directly to the fortigate how does HSRP come into it ? Is the redundancy going to be a manual thing ie. you need to patch the other router in if it fails.

HSRP would surely on work with a common L2 subnet but if the WAN routers are patched directly to the fortigate then unless the fortigate is acting as a L2 switch then how do HSRP messages go between the LAN interfaces on the routers.

So this is the only bit i am not really clear on ie. -

1) are you patching both routers directly to the fortigate on different fortigate interfaces ? - if HSRP doesn't come into it

2) are you only patching one WAN router ? If so this would then require manual switchover and still HSRP doesn't come into it

Jon

Network design query

darren-carr — Tue, 23 Aug 2011 22:34:19 GMT

Hi Jon,

Sorry for the confusion.

I hope this answers you question.

The two routers at each site will be managed by the carrier. It is my understanding that the routers will be connected together by the carrier and this connection is what they will use to run HSRP between the devices.

I am a little concerned about how the Fortigate will detect the link down between itself and the Cisco router. I will need to read up again on HSRP.

I've attached an image regarding what I am trying to achieve. Hope this explains it a little better?

I've attached a

Re: Network design query

Jon Marshall — Tue, 23 Aug 2011 23:51:46 GMT

Darren

Hmm, it depends on how the fortigate reacts if the HSRP active gateway moves to the other router.

There is a an advantage to directly connecting the WAN routers to the fortigates.

That advantage is that because there is no switch between the firewalls and the WAN routers then the fortigate will be able to tell immediately if the interface on the active router has gone down. If there was a switch then you would need some sort of IP SLA functionality on the fortigate to ping the WAN router because obviously the LAN interface of the active router could go down but the fortigate interface is still up because it is connected to the switch.

I'm assuming the ISP will be doing interface tracking with HSRP so if the WAN interface on the active router goes down then the HSRP priority will be reduced and the other WAN router preempts.

So i don't think you need to read up on HSRP. What you do need to work out is how the fortigates react to HSRP switchover. With a switch in between the routers and firewalls the active firewall would simply continue to forward packets via L2 to the new HSRP active router. But there is no direct path from the active fortigate to the new HSRP active router in your design. So i'm wondering how the fortigates react to this.

1) if the LAN interface of the active WAN router fails then that should not be a problem ie. because the foritgate is directly connected then it's interface will also go down so the fortigate and the WAN router should both failover to the standby devices and everything should work as expected.

2) If the WAN interface fails and the HSRP priority is decreased so the other WAN router becomes active how does the fortigate handle this ie. the active firewall only has an indirect connection to the new active WAN router via the firewall interconnect link - can it use this link to get to the WAN router. Even if it can use this link the standby firewall is still in standby mode so would it forward traffic to the WAN router ie. in active/standby usually only the active firewall can forward traffic.

3) It may be that if the WAN interface fails on the HSRP active router then the HSRP priority is not decreased ie. they leave this router as the as the HSRP active and simply route the traffic received from the fortigate active firewall across the WAN router interconnect. This would work.

So you need to understand how the carrier is going to configure these routers and also if they go with 2) how your fortigates will react. I don't think 2) will work, or at least it wouldn't with ASAs because the standby firewall does not pass traffic and there is nothing to tell the active firewall to failover because it's outside interface is still up ie. the LAN interface of the WAN router isn't down, it;s just that the HSRP priority has been reduced so the other router becomes active.

The other problem is return traffic. If return traffic arrives at non-active router, again how does it get to the active fortigate assuming the active fortigate is connected to the HSRP active WAN router. Although again the ISP might be ensuring return traffic always comes down the link it went out on.

Although i said it was an advantage it does seem to complicate issues. This is why a standard setup for this sort of thing is to have a L2 switch or 2 L2 switches interconnected in your case, between the firewalls and routers. Then the active firewall has a path to both routers. With this setup though you do need to be able to check from the firewalls the status of the router LAN interfaces and i don't know if the fortigates have that functionality.

Jon

Network design query

darren-carr — Wed, 24 Aug 2011 00:06:18 GMT

Hi Jon,

Thanks again for the response.

You are correct in assuming that the ISP will be doing the interface tracking with HSRP, so if there was a failure on the Active router upstream then the router would failover over to the standby device. The reason I wanted to read up on HSRP was that I am unclear as to what state the router could then place the interface facing the Fortigate into i.e. could it change the interface status to down? The Fortigate can monitor the physical attributes of its connection to the router.

I will spend some time with the carrier to discuss the desing.

Thanks again for your help. I'll let you know how I go.

Cheers mate

Darren

Network design query

Jon Marshall — Wed, 24 Aug 2011 00:20:08 GMT

Darren

Just for your info, with HSRP tracking the LAN interface would not be shutdown only it's priority decreased enough that the standby router will preempt.

So you definitely need to talk to the carrier about this because i can't see how the active fortigate will get to the new active WAN router ie. the former standby router.

Jon

Re: Network design query

campbell — Wed, 24 Aug 2011 00:57:50 GMT

Hi Darren

I am not sure that the fortigate will give you the fail over you are hoping to achieve. I would assume the provider is giving you redundant IP paths rather than redundant interfaces. That is, HSRP will be presented to you so that you can use the HSRP IP address as your gateway out of your network. If you connect the fortigate in the way, I understand you, then the HSRP will not work and you will end up to two active HSRP nodes.

The fortigate failover mechanism is designed more for two different Internet provider connections rather than receiving one redundant connection. Also, why would you want to limit yourself by only ever utilising one 100Mb connection, when you could design your network to utilise both with a decent routing policy on both ends.

Also, I do not understand why you would separate your VLANS by a layer3 interface on the switches rather than using the fortigate firewalls. By using the switches, you will have to define ACL's per interface and then also configure the firewalls too, whereas, if you are using separation by firewall, then there is only one place for a security policy, although your suggested way would be better if security is not a concern and were hoping to achieve better throughput. I would ask then why then you would split your hosts into vlans, if the latter were true, as there is no real advantage to putting hosts into separate vlans unless you wanted to provide inter-vlan security, in which case, I would suggest you do this with the firewall.

Regards,

Security Consultant

Sent from Cisco Technical Support iPad App

Re: Network design query

Jon Marshall — Wed, 24 Aug 2011 01:05:13 GMT

Darren

Ed said -

That is, HSRP will be presented to you so that you can use the HSRP IP address as your gateway out of your network. If you connect the fortigate in the way, I understand you, then the HSRP will not work and you will end up to two active HSRP nodes.

Apologies but I missed this. I assumed from your diagram that there was an interconnect between the routers. But even if there was this interconnect cannot carry the HSRP hello's for different interfaces on the routers. So Ed is spot on when he says both routers would be HSRP active.

So if the carrier wants to run HSRP you need switches between the routers and firewalls.

Ed - thanks for catching that, i completely missed it which for a LAN switching person is kind of embarrassing

Jon

Re: Network design query

darren-carr — Wed, 24 Aug 2011 01:15:30 GMT

Hi Ed,

Thanks for your input.

You are correct with your first paragraph and I share the same concerns as the Fortigate would see its connection to the router as up even with an upstream failure.

We do use the Fortigate device as you have suggested in paragraph two for our internet connections.

With respect to your query in the secon paragraph it is more about cost. Our standby link is quoted as half the price of the active link.

I now plan to use the Fortigate to do the inter-vlan security/routing. It makes more sense to do it this way. I was playing with the design yesterday and I must admit I wasn't thinking too clearly. It certainly makes more sense and is easier to configure this was as opposed to maintaining ACL's on the layer 3 switch.

I guess for the routing the only option I have is to drop the two router 'inside' facing interfaces into a layer 2 switch, in the same vlan as the Fortigate interface and route to both routers (active/active). I could then split the 100M in two (50M/50M) or double the bandwidth to 100M/100M?

Does this seem reasonable?

Thanks again for your input.

Darren

Re: Network design query

Mohamed Sobair — Wed, 24 Aug 2011 02:27:25 GMT

Hello Daren,

I personally dont recommend/suggest this kind of setup as you mentioned. I dont see a reason of why you make the fortigate perform the Layer-3 routing while you have a capable of Layer-3 Switches that can perform it.

The reason I am saying this is because , you have a switches that perform layer-3 switching even more rapidly than the firewall does because the firewall is primarily not designed to perform this functionality. Its even make more sense if you want to route/filter or apply QoS in the future to have all Layer-3 functionality performed by the 3750 Stack.

It wont increase the config for you, All what you want is to have layer-3 interfaces on the switches for the inter vlan routing and another layer-3 point to point interfaces for the connectivty with the fortigates.

You can stick with the current Setup of having Active/Passive Scenario on the fortigates and One upstream provider active at a time with proper Implementation which should include IP Sla.

You should have 2 default routes from the Stack of 3750 with IP Sla tracking , the active one points to the Primary fortigate while the Backup one points to the Standby fortigate.

With respect to your Internet routers, you should have the 2960 in place as you have drawn in the diagram, and have HSRP Active/Standby between them.

Each of the Internet routers should have 1 default route along with Tracking option , to track its WAN connection and loses its HSRP Active Gateway functionality once it loses its WAN connection and since this is the provider Job , you dont have to worry about it , you can just inform them about this type of confi/implemetation. and I am sure this is what they would perform in the end.

The FOrtigate Firewalls doesnt need 2 default routes, both fortigates should have 1 default route pointing to the HSRP VIP address of the routers and it should point them to the correct Active router based on the previous implementation of the WAN routers Connection and WAN Tracking.

I beleive this is the optimal design for you, and this type of Scenario is more applicable for most similar Setup according to your requirement.

HTH

Mohamed

Network design query

darren-carr — Wed, 24 Aug 2011 04:22:36 GMT

Hi Mohamed,

Thanks too for your input. I appreciate what you have outlined above. I do think that doing the inter-vlan routing through the Fortigate will be easier than maintaining ACL's for the VLANs. The appliance I have is an enterprise class device that is currently doing next to nothing with respect to performance. Managing this through a GUI certainly makes life a lot easier. Happy to be challenged on this though.

With respect to the HSRP operation I am fortunate in that I have a couple of switches in place with a bit of spare capacity to patch the the Cisco routers (inside) interface and Fortigate interface into. The carrier has confirmed this as their preferred solution.

I can easily change to a ACTIVE/ACTIVE solution if performance becomes a problem. The budget and design allows for this. I am just conscious of the costs associated.

Thanks again for all of your input, it has been of great help to me.

Cheers,

Darren

Re: Network design query

campbell — Wed, 24 Aug 2011 09:07:38 GMT

Hi Darren,

I am not sure whether Mohammed understands how the fortigate firewalls' redundancy works, so just to clarify. Fortigate firewalls support an active/passive state, with no traffic being put through the passive firewall at all. Mohammed suggested two default routes from your switches, one to the active and one to the standby. This configuration is not support by the fortigate in active/passive state. You could go with an active/active scenario, but then you would not need two defaults route either, as a multicast mac address is used for the cluster ip address. Also a point to note, some applications like https cannot be load balanced across a fortigate cluster.

To be clear about my inter-vlan separation argument. If security is paramount, and it may not be, then separation by the firewall is the way to go. Switches will not give you stateful checking of packets at all and will be more easy to bypass.

If performance is what you need, then as I stated before, perform separation on the switches then apply access-lists to each interface. However, in my experience and as networks and policies grow, this can be laborious and often forgotten about.

However, if you would tell me the fortigate platforms you are using, then I would be able to judge better whether they will interfere with performance. The fotigate firewall uses ASIC based chips to achieve wire speed firewalling. This is the case with all of their firewalls, so I do not see that performance would be an issue. They also support QoS and bandwidth throttling to the application level. I would hardly think that they will be a problem considering the fact that it is highly unlikely you will have each and every server consuming bandwidth to each server in every other vlan. But as Mohammed suggests, and if security is not a concern, performance through the switches will be better, so you just have to decide on what the risk level is and choose the separation method to suit.

In your last paragraph of your reply to me, the active/active scenario. This method will give you one active, but clustered, IP address to use as you gateway out. So you would have to configure two separate routing interfaces to do what you are suggesting. There maybe a configuration option where it is possible to use an IGP to achieve load balancing over the two links but it depends on equipment at each end and your company's current skill level to support it.

So I would advise this, keep the design simple, make sure that there is the knowledge in your company to support it. If the design is simple then only a few easily identifiable things can go wrong. If it is complicated by various other details, then it will be complicated to support. Look at where you are now and then compare it to what you will have. Is there a need to implement a whole new design, does the business require it? Step back and see what the business requires before embarking on a further design effort. You may find that your needs are far more simplistic.

Good luck, with your design.

Regards,

Sent from Cisco Technical Support iPad App

Re: Network design query

Mohamed Sobair — Wed, 24 Aug 2011 09:37:43 GMT

Hello ED,

I just want to clarify one thing,

The Two default routes from the Stack Switch , is defined with one as primary (Active route) to the Active Fortigate, while the other is ONLY backup (NOT Active). The Backup ONLY Kicks in when the Primary route fails. this what I exactly meant.

Coming to the Security Point, Do you think letting the Fortigate [performs the (Inter-vlan routing) between Local user VLANs adds Security here, I personally see no difference, Since you would still need to add permission on the FW for that, just like when you have ACL on the Switch. its not only the performance is our constraint, but other Options as well.

The firewall would be a good advantage o protect from the Internet as well as from LAN to DMZ and Internet to DMZ , INternet to LAN. but I am not sure, for the LAN segments he needs to pass the traffic through the FW as well.

In Anyway, this is my Openion, and Have seen such similar setup is been done in several situation,

You could have different Openion though.

Thanks,

Mohamed

Network design query

darren-carr — Wed, 24 Aug 2011 10:14:42 GMT

Hi Ed,

My absolute main goal is to make this network design as simple as possible. I don't want to create something that is difficult to manage/administer and every day I work on this I remind myself of that.

With respect to the Fortigate appliance we have a pair of 1000A devices. These are massively over spec for a company of our size (120 users), and were purchased by my predecessor. I've looked at the performance stats and I have little concern about the throughput of these devices. I like the idea of managing the separation of the VLANS through the firewall as it presents a simple GUI, it makes it easier for me to also train those who are working on the appliance. I've worked with ACL's in the past, and although they are effective trying to explain these to others can be quite challenging.

I'm confident that with using the Fortigates as the local VLAN gateway, for each of the vlans provides for a simple design. The WAN will be a managed WAN service so it is what it is. I am trying to make accessing this, via the Fortigates as simple as possible.

I plan to patch 2 x 1G ports from switch 1 in the stack to the ACTIVE Fortigate (Gi 1/0/1, 1/0/2) and 2 x 1G ports from switch 2 in the stack to the PASSIVE Fortigate (Gi 1/0/1, 1/0/2) . I plan to enable port monitor on the aggregated interface.

To review the potential failures:

1. If switch 1 in the stack fails I am confident that the Fortigate cluster will failover to the PASSIVE appliance and traffic will still be routed via the HSRP virtual IP address as the port monitor will detect this

2. If one of the patched interfaces from switch 1 to the ACTIVE Fortigate traffic will continue to route and the cluster will not failover

3. If an upstream connection to the MPLS cloud fails on the ACTIVE Cisco router it will hand the role of the ACTIVE router to the current STANDBY router, this should be seamless to the Fortigate

I have two others sites that will use 10.2.x.x and 10.3.x.x. I plan to create a summary route for each of these networks (10.2.0.0/16 & 10.3.0.0/16) and will route them via the HSRP virtual IP address.

I hope this makes sense?

Thanks,

Darren

Re: Network design query

campbell — Wed, 24 Aug 2011 10:49:09 GMT

Hi Mohamed,

Thanks for the clarification.

However it is a trivial point I think because the firewalls do not give you a secondary interface to route to, unless you create another vlan interface and use it as your secondary route. In active/active you have one shared virtual IP and in active/standby you have one IP that's fails over to the other node. There is no need for a secondary route as you suggest.

On the security point you make, there is a whole lot of difference. One is a switch with no security features at all and the other is a firewall that checks each packet with stateful inspection and provides a whole range of security features that the switch can't.

I think you are missing the point, the design should be simple and easy to support. Should Darren leave the company tomorrow, then his underlings will still need to understand the design. I would think that the switch Option here would complicated design when there no need.

Sent from Cisco Technical Support iPad App

Re: Network design query

campbell — Wed, 24 Aug 2011 11:13:11 GMT

Hey Darren,

Yes I see how you are doing the failover. The fortigate will support this without issue. You should look at trying to allow the active firewall to always be online, so perhaps this may help:

Fw0-int-0 -> sw0

Fw0-int-1 -> sw1

Fw1-int-0 -> sw0

Fw1-int-1 -> sw1

If you replicate this for outside and inside firewall, it should survive one side of your network going down and still route to the HSRP IP with the active firewall still active. The stack will allow this configuration, it is a standard configuration I use for existing clients. a switch failure will mean nothing to the firewall as it will still have an active interface on the inside and out. Just test the aggregate failover feature, to make sure it doesn't fail the firewall too if one interface of the group fails. You should be able to control how this happens anyway.

You are correct about the 1000 series firewalls, they are overkill but that puts you in an even better situation. The performance will not be a problem at all between networks, but just be aware of how many vlans you put on one interface. If there are 10 vlans, then they all share the one gigabit port etc, you get the idea.

What routing protocol had you planned to use to advertise your summary routes? Or were you just planning to use static? Since there is only one way out of the network, static should be sufficient.

Don't forget to do a good diagram too, so that others are aware how things are connected and how they fail over.

Regards,

Sent from Cisco Technical Support iPad App

Re: Network design query

darren-carr — Wed, 24 Aug 2011 11:47:11 GMT

Hi Ed,

Just to clarify something here....

Lets say my appliances are FW1 and FW2 and I have SWITCH-1 (first switch in stack) and SWITCH-2 (second switch in stack)

I currently only have three interfaces available until I move the data centre off site, those interfaces are port6, port7 and port9

What I was thinking was creating a 802.3ad interface made up of port6 & 7 and use port9 for the WAN interface (10.1.255.1/29)

Lets also say I have switchports Gi 1/0/42 & Gi 1/0/43 available on SWITCH-1 and Gi 2/0/42 & Gi 2/0/43 available on SWITCH-2

If I therefore create a port-channel made up of Gi 1/0/42 & Gi 2/0/42 and physically patch these interfaces to FW1 port6 & 7 this would be the optimal configuration. This should ensure that if SWITCH-1 or SWITCH-2 fails that FW1 remains active. I'm pretty sure that this is what you were stating? Whilst using 1/0/43 and 2/0/43 for FW2 port6 & 7.

With respect to the WAN interface (port9) I was planning on configuring this physical inteface with ip address 10.1.255.1/29 patching this interface (from FW1 & FW2) into VLAN 255 that is configured on two Cisco 2960 layer 2 switches that are connected using ports Gi 0/47 & 48. I was then planning on patching the 'inside' facing interface of each of the Cisco routers into the same VLAN. Allocating the ISP the IP address 10.1.255.2/29 for the virtual IP address and .3 and .4 for the physical IP addresses.

I would then route anything off the local VLANs, with a static route, to 10.1.255.2?

Does this sound reasonable?

I do like to back all these things up with a diagram, it is one of my key KPI's so I have to!!!

Cheers,

Darren

Re: Network design query

campbell — Wed, 24 Aug 2011 11:56:07 GMT

Hey Darren,

Yes, that's exactly what I was saying.

Also, if possible, configure your wan interface as an aggregate of one, this will allow you to add another later when you have freed an interface after your move to the datacenter, and will give you the same level of redundancy as the inside.

The rest should be plain sailing.

Regards,

Sent from Cisco Technical Support iPad App