ISP Infrastructure Setup?

[CiscoPurpleBelt]

Trying to get more insight about ISP infrastructures. So Im familiar with basic BGP, MPLS, OSPF setup. Question I have is are the PE and CE routers are installed in other regions lets say Verizon, Att, etc data centers in different regions e.g New York, California,  etc.? Typically about how many devices would you find in a Core distro point Data Center?

[Joseph W. Doherty]

How many core device in a data center? Depends on the size of the data center and gear being used. Today, if you're using fabric devices your core comprises what ever number of device you believe you need for performance (starting, generally, with two).

Unclear what you're asking about "other regions" and PEs and CEs. CEs, of course, are often on customer premise. PEs might be located in some local cluster and often have multiple customer CEs connected to them.

[CiscoPurpleBelt]

Great thanks!

Sorry to provide additional info, let's say CE is in California. PE may be in let's say Nevada and the Core Provider routers (rest of mpls enabled routers) be in a provider data center in another state let's say (given it is a transport or IP provider running mpls/bgp, etc.)?

Perhaps a better question would be what compromises an ISP infrastructure let's say like the size of Verizon (e.g. 100 Core routers, 200 PE routers, 10 load balancers, etc. all in 1 or the same data center)? I am just trying to get a better picture of the general architecture involved.

[Joseph W. Doherty · Accepted solution]

Cannot say for sure about Verizon, as I've never worked for them, but having worked for another very, very large ISP (and cable video) provider, we didn't have too many really huge data centers because the data centers were for our business, not the public at large.

As to the network itself, the backbone "core" routers were very large routers (like CRS-3s, or Brand "X" equivalent, or better) connecting our US backbone links (100G). Those routers, in turn, connected to large routers in various multi-state regions. Those too might have 100g links, or 40g links or Etherchannels of 10g links. Those routers then connected to regional distribution routers (sometimes the larger ASR 9000s, or equivalent Brand "X"), again possibly with 100g or 40g links, or more commonly one or more 10g links. The regional distribution routers would often, besides other purposes, might fulfill the role of PEs.

I didn't support our data centers (a different network engineering group), but I believe they used large Nexus switches, with possibly 100g connections into the regional network. I believe they use the Nexus spline/leaf architectures, so again, they could scale devices pretty much as desired.

I didn't support our backbone, or other "public" infrastructure networks, I supported our Enterprise network, which although wasn't the "bread and butter" (i.e. a profit center) of our business(es), was still a large network itself, as we supported over 100,000 employees/contractors using it.

[CiscoPurpleBelt]

Awesome info! So basically, some ISPs backbone routers in one location may very well be let's say all racked in one 5 - 1- rowish enclosure isle?
When a place says amongst the largest ISP in the world, I was not sure to think of IDK like 100+ enclosures of all backbone CRS type routers or just may 10 PE routers in one location, etc. I just have read Cisco docs and used labs of 30+ routers however I don't really know what to really expect in the real world.