Collapsed core -> Traditional

[johnelliot6]

Hi Guys,

We have 5 POP's, each running 7200's (G1's + G2's), that connect to 3750's/2960's - We have a bunch of carrier connections(Trunk ports) coming into

the switches(client tails handed off as vlans) and also atm(into the 7200's) for dsl.

The 7200's are P/PE's running MPBGP/VRF's/MPLS/OSPF/Inet/Netflow - i.e they do all the L3....the switches are only L2

Couple of our pop's are pushing 450Mb+/sec aggregate traffic, and the G2's are getting up to 50% cpu on busy days(So need to look at upgrading)

Probably the "easiest" upgrade path is to replace the 7200's with ASR1000's, but I would like to also look at potentially going to a more traditional design.

Know it's a very complicated question, but is moving to 4500's/6500's as "core", with 2960's/3750's as access and using the 7200's for purely edge(Inet/LNS) duties a better approach?

Any feedback is greatly appreciated.

[Edison Ortiz]

You are right, it is a complicated question w/o looking at the entire picture which will take more than one posting.

The general guideline is to have a powerful switch at the core/distribution segment. That is, if you are doing a lot of intervlan routing and the need for redundancy.

I won't mention the 2960s since they don't provide the redundancy that is needed so let's focus on the 3750s.

3750s will provide redundancy and port density but does it provide enough throughput for your needs?

If not, then we need to explore the 4500 with Sup7-E with 6500 with Sup2T.

It's your call in terms of pricing and support, when you break away the layers (access, distribution, wan) you have a more scalable network but that comes with a price tag. You also eliminate single point of failures in your network.

[johnelliot6]

Thanks for the reply Edison - much appreciated.

If we are to look at 4500's as core, and as we do a lot of client vrfs, can the 4500's perform the role that the 7200's are currently doing(Aside from the upstream Internet connectivity and LNS duties) - i.e. Our 7200's have hundreds of dot1q subints, that are placed into vrf's or "Internet"...or does L3 stay on the 7200's?

Cheers.

[Edison Ortiz]

You should definitely move the L3 routing to the Core 4500. With the 7200, you are limited to the interface capacity which I assume is 1Gbps whereas the 4500 can provide up to 848Gbps switching capacity when using Sup7-E.

[johnelliot6]

Thanks Edison - To confirm (I have searched for documentation on Cisco/Google, but nothing I found really answered my questions!)

Our current roll-out(For new pop) would consist of:

Carrier tails terminate on 3750/2960 as trunks (vlans presented as individual client services), we then trunk to our 7200 and these vlans are setup as portchan dot1q subints (In either a vrf, or "inet")

The 7200 would then have ospf setup on "interpop" links(Back via the 2960/3750 as this is where the carriers are terminated), mpls enabled on those links, then ibgp mesh configured on "new" 7200 and all other 7200's in our network.

So, to "replace" our 7200's with 4500's, would we still setup the 4500's as we currently do the 7200's?(ospf/ibgp/mpls)....but the "portchan dot1q subints" would be configured as vlan interfaces on the 4500's?

Apologies for the rather elementary questions, but I've never touched a 4500/6500 

[Edison Ortiz]

Based on your description, the sole function for these 7200 routers is routing. I was assuming they were internet/wan edge device but it seems they are not, correct?

Additionally, the goal is to keep the customer routing totally isolated from each other, correct?

The 4500 supports VRF lite and you replace the subinterface concept with 802.1q trunks (same design concept you are using today with the 3750s). You can also keep your iBGP mesh and OSPF.

When you mentioned MPLS, are you running MPLS on your routers? or just VRF lite?

If you need MPLS PE functions, you need a 6500 with Sup720.

Regards,

Edison

[johnelliot6]

Hi Edison,

Yes, our 7200's do all of our routing (As well as taking external bgp peering sessions for Internet Access) - They are P/PE's.

We are running MPLS on all of our 7200's (On the interpop links), we do not run vrf lite.

Example scenario for a client:

Has ethernet tails terminating on POP A, POP B and POP C - We place those tails on the 7200's in VRF CLIENT_A, then as all of our 7200's are in IBGP mesh, the clients "sites" can reach each other.

So, yes for clients that have a VRF service, we most definitely want to keep routing totally isolated.

Based on the above, it seems the 4500 would not do what we want, and we would need to go to a 6500?

Thanks again for your help

[Konstantin Dunaev]

Hello,

it sounds like we have a similar topology

But we built a little bit different:

- each PoP has one c7200 (edge router) which provide external BGP peering and at the same time is a "P" router for our MPLS

- each PoP has a couple of 3650G switches, where we're terminating all Ethernet client tails : physically and logically (each customer has a separate VLAN and  configured L3 VLAN interface). c3560 are terminated onto c7200.

- some PoPs has "service" router like for DSL termination, IPSec VPN, MPLS PE routers and so on. all those routers are terminated onto  edge routers

- some PoP has only a couple of c3560G and are connected to a next PoP with a c7200

we have a very little traffic which stays inside of our network, most of it goes outside, it means there is no much cense (for us) to create an additional level of aggregation (like gather all c3560G onto one pair of c6500) because anyway the whole traffic will go through c7200 outside.

But in your case I'd suggest at least to move the "customer-tail" interfaces onto separate devices (onto 3750 ? ), it' become more structured any logically separated into "customer-tail" and "backbone".

[johnelliot6]

Hi,

Yes, our topologies do sound similiar 

To move the customer tails onto 3750 (And place them into vrf's as we currently do), how does 3750 at POP A, know that 3750 at POP B has a tail in the same client vrf(eg CLIENT_A vrf)? Would we need to create a seperate interface (POP A->POP B) that would need to be also placed in the clients vrf(CLIENT_A vrf), so the 2 sites could communicate?

If so, we would lose all redundancy between our POPs (We have multiple links between them, that have different ospf costs to accomodate for link failures)....unless there is some other method with vrf lite?

Thanks for your response.

[Konstantin Dunaev]

We don't use VRF-lite exactly because of those reasons you've outlined.

In our case, we place in a necessary PoP (actually in all PoPs with an "edge" router and upstream uplink) a separate router where we are terminating a MPLS customers (via subinterface from a c3560G, where the customer has a port), those routers becomes a PE routers and are connected via P routers in a  MPLS backbone.

All normal "internet" customers get an interface on a c3560G. 

Each c3560G and PE router are connected to an edge router in a local PoP and to one of near PoP, for redundency.

What I mean, if you have a lot of traffic which is forwarded between your PoPs (MPLS-VPN traffic or something like that) then it would be a good idea to put in some PoPs  a couple of high end routers or L3 switches (with full MPLS) as an aggregation point for MPLS traffic and get those traffic away from your edge c7200. Or as a second possibility you can  put someting like ASR 1000 instead of your c7200's and use them as P routers for MPLS and internet traffic without agggregation level.

But if you don't have much inter-PoP traffic then you need just think about upgrading your edge routers and you don't need additional aggregation level, you're already aggregating your customer on c3750.

[johnelliot6]

Thanks for the reply Konstantin.

I have looked at the ASR1000's (And that is what I was originally going to do....simply replace the 7200's with ASR's), but as we do quite a bit of "inter-pop" traffic, it might be a better option to retain the 7200's and use them purely for LNS and external bgp(Internet) devices, and have 6503's as P routers?

[Konstantin Dunaev]

in your case to put c6500 in aggregation level and use them as P/PE router seems to be reasonable.  But take into consideration, that if your external traffic grows up to 800-900 Mbit on a single PoP you c7200 NPE-G2 should be updraged too and  c6500 has some limitations  , mainly it's the small Netflow table which doesn't allow to get full Netflow flows on a heavy loaded interfaces .

[johnelliot6]

Thanks Konstantin - So based on my setup, what is required for a 6503/7603 (I would want to go dual sups)....we do some qos currently(7200->client CE match markings, and apply priorities, and also some bronze/silver/gold priortisation on inter-pop links), mpls/ibgp/ospf...Ive done some reasearch on both(7600/6500), and it looks like the 7600 is more appropriate(more wan cards?), we also rely on netflow(For billing).....the 6500's and 7600's certainly arent as "simple" as the 7200's when it comes to feature requirements!

btw - our external traffic is ~100Mb so we have some breathing space.

Cheers!

[Konstantin Dunaev]

if you want to be more flexible about WAN interfaces then 7600 is a better. But 7600/6500 both use the same supervisor and the feature set is  pretty the same.  Netflow could be a bottleneck, but only if you have a huge amount of traffic and would like to see full Netflow.

May be you should have look at a different MPLS features which are available on 6500/7600, e.g. Ethernet over MPLS, but it  requires a special linecard, AFAIK.

I had quite good experience with c6500's and can't say  anything bad, except the price

[Edison Ortiz]

Per your requirements, the 6500 will be a very expensive solution. If port density is not a requirement, I suggest investigating the ASR1k line.

For advanced QoS in the 6500 line, you need a WAN module (SIP/SPA) if you go with Sup720.