Network design - apologies for the detail!

[darren-carr]

Hi All,

I am currently working on a design for our new network. We currently host our Production data centre on premise, with the business. We have approximately 120 users all based on the one site. We also have a DR site that is around 15kms from the business. The current network design is very flat and I want to make use of Cisco best practices to re-architect the network.

Some further information may help you understand my thoughts/design:

- the business is based over two floors

- the business premise is soon to be refurbished with users now being spread 70/30 over the two floors.

- we have 120 users in the business

- we are also looking to roll out a new IP Tel solution and wireless network

With this in mind (and you can see in the design) I have selected switches from the 3750-X range to support both the access and distribution/collapsed core at the business premise. The reasons for doing this are:

• •1) The switches are stackable
  
• •2) We may chose to route from the access layer in the future
  
• •3) They have dual power supplies (we need maximum redundancy for our business as we are a financial institution)
  

With us moving the data centre off-site we are also looking to upgrade our WAN. We have decided to outsource the management of the WAN to a local, reputable, ISP. The WAN is based on a private MPLS VPN cloud offering and the access rate is scalable up to 1Gbps with no significant hardware changes. The routers they plan to use for this are Cisco 3845’s. We will have two at each site to provide a fully redundant network. We will also have complete diversity on the fibre to the three sites as the carrier supports complete path diversity. The carrier will use HSRP between the two routes at each of the sites for logical redundancy.

We have around 100 servers at present, a mix of physical and virtual, with the majority being virtual. We currently have 6 racks that are fully populated with servers (one rack is a large SUN M5000 investment). The rest of the network is dual attached to a stack of Cisco 3750G switches. The plan is to deploy TOR switches in the form of the Nexus 2248TP (12 at day one – six racks) uplinked to the Cisco Nexus 5548P (2 at day one).

Our operations team have complained in the past about an overly complex IP addressing scheme and have requested that we deploy addresses in the 10.x.x.x network at each of the three sites. So this is how the addressing will be laid out:

- the 10.1.x.x subnet is where the business will be located
- the 10.2.x.x subnet is where the Production data centre will be located
- the 10.3.x.x subnet is where the DR data center will be located
- at the production data centre we will have 6 racks with Nexus 2k (FEX) deployed in and the core comms will have 2 x Nexus 5548P devices (still working on how to connect these devices)
- on the routers at the 10.1.x.x site, I plan to configure a VLAN interface on the router with a /29 address (2 x ip for routers, 2 x ip for Nexus 5k, 1 x ip for firewall)
- users at 10.1.x.x need to be able to access the Prod and Test servers at 10.2.x.x
- users also need to be able to access servers at 10.3.x.x
- the firewall cluster has been deployed at the 10.2.x.x site to control inbound access to our DMZ, etc

If you look at the design I have a couple of queries regarding the placement of devices in the network. I am comfortable with the access layer at the building premise. There is however a need for the following:

1) Restrict access, to IT developers, to the Production server environment. The obviously need to access Production servers for email, applications, etc but we need to restrict access to other servers and prevent them from adjusting configurations. They should be able to get to the test environment. Reviewing Cisco’s best practices it would suggest that these restrictions should be placed as close as possible to the device (developer). I was therefore thinking at the distribution layer. I therefore see EACL’s as the only option with this design, unless I install a firewall infront of the WAN routers (3845’s)? This would stop traffic needlessly traversing the WAN?

2) I also need to do this for users accessing the DR site. I am thinking the same again as the above for achieving this.

I’d be interested to hear your thoughts on this? The design is still a work in progress. Please also feel free to comment on anything else you would consider changing?

Thanks,
Darren

[Robert W. Rogier]

Darren,

     I will let you read my profile so you'll know where I'm coming from.  To put it shortly though, I have been the network architect and have done network design and operations for the better part of 10 years.  I know there a likely a number of battles going into this concerning design, accessibility, and most of all, cost.  All that being said, I think I would make a few updates to your design.  Thank you very much for the visual, it helped a great deal in your explanation.  I do have a couple of questions though that will influence the design more, but you can answer those after this.  The questions are:

     1.) Is there a need for remote access to any location and how is this to be provided? (Modem, VPN, etc)

     2.) Are there prevailing laws that are going to require the encryption of some if not all traffic on the LAN/WAN?

     3.) Does the IP Telephony system you are looking at take into account any encryption necessary.  I ask this because often encryption is kind of an after thought to VoIP so getting it in at the beginning is critical.

     Those asked, here are my recommendations/changes:

     1.) Business

          a.) You stated you were specifically using these switches because they are stackable.  When I look at the network design you have given, it appears that the 3700's are just simply connected with no stacking.  The design look for stackable is to have each switch with on connection to the other and then a final connection from the top of the stack to the bottom. Once you have this, pick either 1, 2, or 3 ports on each individual switch and connect back to the core.  I would recommend picking 2 for this reason.  The likely hood of losing any two switches (without also losing the third) is infinitesimally small).  The stack-wise cables give you complete redundancy to the stack allowing you to just need 2 connections to the core from two different switches.

          b.) I would make the classic bounded x configuration from your cores to the WAN routers. This is to say each core switch connects to each WAN router and vice-versa.  The method you have could leave you some exposure if you were to lose say the top switch in the aggregation/core area.

          c.) Security -- This is the one topic you didn't discuss.  I assume you are implementing firewalls off of each Internet connection.  I would suggest you go one step further and solve your issue with needing eACLs on the switches.   Place two small ASA5508s just before the core/aggregation.  You could also use these at each remote data center (and I would urge this.)  You know the type of data that should be coming into your data center.  This just keeps everything on the up and up with the ISP.

     2.) Other locations.

          a.) I'm a firm believer in security that is frequent and has a default closed policy.  Using ASA's spread throughout the network will give you traffic control over each section and give you very good auditability.

          b.) I also wonder if you have taken into account any DMZ servers that may be necessary as well as an IDS system.  All of this requires planning ahead of time.

     I could literally go on for pages and pages on this.  A few things that should be thought about during this phase beyond the above is this:

     1.) QOS Implementation.  This is the heart of VoIP and needs to be done right.  Be sure you have adequate design around this.

     2.) Overall Management.  Be sure that in your cost to implement is some tool to manage all of this, even if it's Cacti and Nagios at first.  Having all this power and security is great, but if no one reads the logs, you're sunk.

     Please let me know if you have any other questions.  I hope this has been helpful.

Regards,

Robert W. Rogier

CSE -- TAC UCC

[darren-carr]

Hello Robert,

Many thanks for your input. I appreciate all of your comments.

With respect to your queries:

1) Yes there is a need for users to VPN back into the office. We have a very legacy and rather cumbersome way of providing this access which is soon to be addressed. User VPN connections are terminated on the firewall, and will cross the WAN from the data centre back to their individual workstation. This solution is soon to be replaced with a terminal service solution hosted out of the two data centres (Production & DR).

2) The ISP WAN is a secure, MPLS VPN that will connect the three sites.

3) The IP telephony solution, that is set to be deployed in the next 18 months, will be deployed over separate, managed infrastructure. There is a business requirement to keep data and voice physically separate.

The 3750's at each site will be stacked for sure. There will be three stacks of switches at the 10.1.x.x site, one at the 10.2.x.x site and two at the 10.3.x.x site. The 10.3.x.x site will make use of existing 2960 switches, top of rack.

Point taken on 1-b), a good point I will speak to the carrier regarding this.

With respect to security, on the 10.1.x.x network there will be a small user presence. We need to manage security about L3/4 here and have a significant investment in ISA. Very likely that this will remain here. We also will make use of existing Fortigate devices at the data centre sites to separate the various networks. We are looking to outsource some of the services we offer through our DMZ to a managed service provider (messaging, email scanning, etc).

QoS will be implemented on the data WAN. We are working with the carrier on this to define out polcies, etc.

Management is a big concern for us. We are also actively working on this. May I ask what product you use to manage your Cisco environments?

I also have another question if I may. I will PM this to you.

Thanks for taking the time to reply.

Darren