cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Community Tech-Talk Carrier Routing System (CRS) Multichassis

5929
Views
30
Helpful
3
Comments
vinayk3
Cisco Employee

I am Vinay Kumar (CCIE R&S #35210) and I work for the High Touch Technical Support (HTTS) team, a team that provides reactive technical support to majority of Cisco’s premium customers.

My team and I have been providing support to leading Service Providers and large Enterprise customers for many years. As majority of these customers are transitioning to XR platforms in their network, we wanted to discuss about the HW Architecture of Carrier Routing System (CRS). Rahul Rammanohar (CCIE R&S, SP #13015), Hitesh Kumar (CCIE SP #38757) and I have created this blog and a video discussing the same.

Everyone today is talking about the Next Generation Internet and Communication services with major focus on Cloud Technologies, Data Centers, Video and Media rich applications which has increased the demand of Bandwidth and has created capacity challenge in core for Service Providers. This is where Cisco’s Carrier Routing System (CRS) comes in to picture which delivers continuous, always-on operations and scales easily from numerous single-chassis form factors to a massive multi-chassis system. Its highly efficient design conserves power, cooling, and rack-space resources while optimizing bandwidth capacity.

We already have CRS routers available in two form factors CRS-1 and CRS-3 and next Generation CRS-X coming soon.

CRS routers are available in 4 slot, 8 Slot and 16 slot chassis and can be used either in Satnd alone mode, form a Back to Back system or can be used in Multichassis environment.

CRS_Tech_Talk.PNG

Before we start explaining about the CRS Multi chassis architecture let’s take a quick look at CRS Single chassis router.

CRS Single Chassis:

A CRS router with all the Line cards, Fabric cards which implements all three stages of Switch fabric and Route Processor cards in Single chassis makes a single shelf CRS system. The system uses Cisco IOS XR Software, designed for always-on operation. Cisco IOS XR Software is the only fully modular, fully distributed internetwork operating system using a memory-protected, microkernel-based architecture and control-plane distribution that allows the system to scale.

The Single -shelf system is fully compatible with existing and future components of the Cisco CRS Family, such as router processors and line cards.

The fully redundant carrier-class configuration supports in-service upgrades from 40 Gbps per slot and 140 Gbps per slot to 400 Gbps per slot, when available, and from a single-chassis to a multichassis system.

CRS Line cards:

MSC ( Modular Service Card ),FP ( Forwarding Processor ) and LSP ( Lable Switch Processor ) three forms of Line cards or Layer 3 forwarding engines are available with CRS routers. Each MSC, FP, and LSP is paired with a corresponding physical layer interface module (PLIM) that contains the physical interfaces for the line card. An MSC, FP, or LSP can be paired with different types of PLIMs to provide a variety of packet interfaces.

We have three types of MSC cards CRS-MSC (End of Sale) , CRS-MSC-B & CRS-MSC-140G, Two types of Forwarding Processors CRS-FP40 & CRS-FP-140 and One LSP CRS-LSP card available which can be used with variety of PLIM cards.

Single chassis CRS routers also have the Fabric cards CRS-16-FC/S (40G) & CRS-16-FC140/S (140G) Which is responsible for providing the three stage Switching. If the line card chassis is operating as part of a multi chassis system, the chassis accepts either the CRS-16-FC/M (40G) or the CRS-16-FC140/M (140G) SFC.

CRS routers have variety of Route Processor cards available as well which are mentioned below.


  • CRS-8-PRP-6G
  • CRS-8-PRP-12G
  • CRS-8-RP
  • CRS-16-PRP-6G
  • CRS-16-PRP-12G
  • CRS-16-RP
  • CRS-16-RP-B

CRS-8-PRP can be used with both 4 slot and 8 SLOT CRS chassis.

CRS Switch Fabric Overview

The core of the CRS routing system is its Switch Fabric system created by multiple redundant Switch Fabric cards. The switch fabric receives user data from a modular services card (MSC) or Forwarding Processing card (FP) and performs the switching necessary to route the data to the appropriate egress MSC or FP. switch fabric uses a cell-switched, buffered, three-stage Benes switch fabric architecture. The Complete Switch fabric is logically divided in to 8 (0-7) independent, identical and unsynchronized fabric planes and each fabric plane is comprised of S1, S2 and S3 Asics. Several switch element components on each switch fabric card perform the functions to implement each of the three stages (S1, S2, and S3) of the switch fabric. Each stage performs a different function.

Stage 1:- Stage 1 elements receive cells from the ingress MSC and PLIM (or RP) and distribute the cells to Stage 2 (S2) of the fabric plane. Cells are distributed to S2 elements in round-robin fashion; one cell goes to the first S2 element, the next cell goes to the next S2 element, the next cell goes to the third S2 element, and so on, and then back to the first S2 in sequence.

Stage 2:- Performs switching, provides 2 times (2x) speedup of cells, and performs the first stage of the multicast function. Stage 2 elements receive cells from Stage 1 and route them toward the appropriate Egress MSC and PLIM or Egress Line card chassis in Multi shelf system.

Stage 3:- Performs switching, provides 2 times (2x) speedup of cells, and performs a second level of the multicast function. Stage 3 elements receive cells from Stage 2 and perform the switching necessary to route each cell to the appropriate egress MSC and PLIM.

In a CRS Single Chassis this three stage fabric Operation is performed by a single Fabric card S123. Each S123 switch fabric card contains two S1, two S2, and four S3 elements.

More details about the Switch fabric architecture are covered in the later sections.

CRS Multichassis

As mentioned earlier a Fabric cards in a Single chassis CRS implements all three stages of switch fabric in a single card, however we can take the stage 2 out of this fabric card and implement a multi chassis CRS router. Scaling from a single-chassis to a multi chassis system allows service providers to expand their network without increasing the complexity of their routing architecture

CRS multi shelf routers essentially have two main components.

  • Line Card Chassis ( LCC )
  • Fabric Card Chassis ( FCC )


Line card Chassis :-

A Line card chassis in CRS multi chassis system consists of 16 slots for modular services cards (MSCs), forwarding processor (FPs) cards, and label switch processor (LSP) cards (all referred to as line cards); associated physical layer interface modules (PLIMs) and SPA Interface Processors (SIPs). It also hosts two route processors ( RP ) / Performer route processor ( PRP ) cards and a eight Switch fabric cards which provided first and third stage of switching known as S13 card. So stage 2 or S2 switching operation is not available in Line card chassis in multi chassis system. The LCC supports both 40 G and 140 G fabric cards and line cards. The Cisco CRS-1 Carrier Routing System uses fabric cards designed for 40 G operation (CRS-16-FC/M cards) and the Cisco CRS-3 Carrier Routing System uses fabric cards designed for 140 G operation (CRS-16-FC140/M cards).

Fabric Card Chassis:-

Major Component of multi chassis system are Switch fabric cards (SFC) that enables packets to be switched from source to destination. The SFCs on the FCC provide Stage 2 of the three-stage Benes switch fabric for the multi chassis system. The S13 SFCs in the LCC provide Stage 1 and Stage 3 of the switch fabric. Two types of SFCs are supported: CRS-FCC-SFC for the 40 G CRS-1 system, and CRS-FCC-SFC-140 for the 140 G CRS-3 systems. Either eight or twenty-four SFCs are needed, depending on vertical or horizontal cabling. SFCs are located at the front of the FCC.

Optical Interface Modules (OIMs) are passive devices that provide the fiber cross-connect function. The OIMs distribute the fibers within each cable bundle to the SFCs. Each OIM is matted to an SFC. The OIMs are monitored by the OIM-LED card. Each OIM has 9 interfaces and either 8 or 24 OIMs are needed, depending on vertical or horizontal cabling. The OIMs and cables are located at the rear of the FCC

Another important component in FCC is Integrated Shelf Controller Gigabit Ethernet (SCGE) cards. The 22-port SCGE card (CRS-FCC-SC-22GE) serves as a shelf controller for the FCC, providing the control function similar to the RP for LCC. The 22-port integrated GE switch provides the connectivity for control protocol between the FCC and LCC. Two 22-port SCGE card is included in each FCC for redundancy. Only one shelf controller card is active at a time. The second acts as a "standby" shelf controller, serving as a backup if the active card fails. SCGE cards located at the front of the chassis.

Switch Fabric Architecture:-

As mentioned earlier Switch fabric is the heart of CRS router and it’s a 3 stage switching and logically divided in to 8 planes. Now Let’s take a look how Switch fabric is connected with Line cards and Internally.

Below image shows the logical connectivity of the Switch Fabric with Line cards.

Switch Fabric.jpg

The ingress line card receives packets from the wire. It segments the packets into 136 byte cells and sends these cells to the stage 1 or S1 of the switch fabric. Each cell is always sent on one fabric plane only. Each cell includes a 12 byte header, a 4 byte ECC and 120 bytes of data. In a CRS 16 slot chassis, line cards 0-7 send the cells to the top S1 ASIC only whereas the line cards 8-15 and the Route Processor send the cells to the lower S1. The cells are then evenly distributed to the S2 ASICs in a round-robin fashion. The S2 ASIC performs switching based on the 12 byte cell header and directs the packet to the appropriate S3 ASIC. It also performs the first level of multicast replication. The S3 ASIC receives the cell from S2 and routes the cell to the appropriate egress line card. It also performs a second level of multicast replication if needed. On the egress side we have a total of 4 S3 Asics per plane, 2 switch traffic for slots 0-7 and the other 2 switch traffic for slots 8-15 and the RP’s.

Each ingress line card has 4 connections to each fabric plane which adds upto 32 connections to the S1 ASICs. Each connection is of 2.5Gbps. however the bandwidth available for packets is reduced further as we use 8b/10b encoding and a 16 byte cell header for every 136 byte cell. The bandwidth is approximately 56 Gbps.

The RP’s bandwidth is a lot lesser. In a 16 slot chassis the RP has 2 connections to the lower S1 asic in each plane with each link being 2.5Gbps so total bandwidth is 16*2.5 gbps = 40 Gbps. After the 8b/10b encoding and cell overhead, the effective bandwidth reduces to 16 Gbps.

Each black line between the S1, S2 and S3 represent 18 links of 2.5Gbps. So, each S1 sends cells to each S2 via 18 links and each S2 sends cells via 72 links to the S3 ASICs. Similarly, each S3 asic would receive 36 links, 18 from each S2 ASIC.

Moving to the links between the S3 ASICs and the egress line cards. There are 4 connections from each S3 towards the egress line card. There are a total of 8 planes, hence we have 64 connections. As each link is of 2.5Gbps, the total bandwidth would be 160Gbps. But due to the 8B/10B encoding and cell overhead, the effective bandwidth reduces to approximately 112 Gbps.

Also, each linecard receives 8 transmit links per S3 pair and the RP receives 4 Transmit links per S3 Pair.

In CRS-3 We have 2 S1, 3 S2 and 2 S3 Asics. In CRS-3 all Line cards and RPs are connected to both S1 ASIC’s. From Each line card there are 3 connections to each S1 Asics, which means in CRS-3 router from LC to S1 there are 6 links so with total of 8 planes we have 48 links out of which 40 links carry the data traffic and each of this link is of 5 Gbps so we have total Bandwidth of 200 Gbps. After the encoding and cell tax deductions we get approx 140 gbps.

In CRS-3 there are 72 links from S1 stage to the 3 S2 ASICs (12 links from Each S1 to S2) and 144 links from S2 stage to the 2 S3 ASICs (24 links to each ASIC).

On the Egress side we have 2 S3 Asics per line card. Each S3 asics receives 72 out of 144 links in 16 slot chassis. Similar to CRS-1, we have 64 connections in total between egress LC and S3 but each link here is 5 gbps so total bandwidth is 320 gbps and after the same deductions we get 225 Gbps.


Configuring the Fabric Planes:-

In a multi chassis CRS router Fabric chassis has 24 slots for S2 fabric cards which are divided in 8 fabric planes. Planes are not tied to any specific slots we can manually define which cards will form a specific plane. Below is a config example:

controllers fabric plane 1

oim count 3

oim instance 0 location F0/SM8/FM

oim instance 1 location F0/SM7/FM

oim instance 2 location F0/SM6/FM

OIM count 3 means The cables from each LCC for that plane connect to different OIMs and a count of 1 means All cables in plane connect to the same OIM.

From the above config we see that S2 Cards in SLOT 8, SLOT 7 and SLOT6 are part of fabric plane 1.

CRS Control Ethernet:-

As mentioned earlier FCC has a two 22-port Shelf controller gigabit Ethernet card (CRS-FCC-SC-22GE) serves as a shelf controller for the FCC, providing the control function similar to the RP for LCC. This card has the integrated switches which helps in communication with the Line cards RP’s.

The switches providing connectivity inside the rack via FE ports are referred to as ‘intra-rack’ switches. These switches are similar to the ones used on the RPs and the switches which provide connectivity between racks are referred to as ‘inter-rack’ switches. The distinction is important to know as both kinds of switches are present on the same board and the CLI look and feel is similar.

We have 2 SCGE cards in the FCC one of which remains active and other remains standby. One of these two cards needs to be operational otherwise FCC will shut down.

All communication from the Line card RPs to integrated switch is over the Control Ethernet which is also responsible for some other important functions like system boot up and node availability check by using the heartbeat messages.

The Control Ethernet is redundant and must be connected in a fully meshed configuration to all active and standby RPs and SCs. For example in a 2+1 multi chassis system ( 2 LCC and 1 FCC ) to have full mesh we need total 9 cables , 8 RP to SCGE and 1 SCGE to SCGE. Spanning tree protocol is used to determine which paths will be used for communications.

Below command output show a sample output verifying the control Ethernet connectivity.

RP/0/RP0/CPU0:router(admin)#show controllers switch 0 ports location 0/RP0/CPU0

Ports Active on Switch 0

FE Port 0 : Up, STP State : FORWARDING (Connects to - 0/RP0)

FE Port 1 : Up, STP State : BLOCKING (Connects to - 0/RP1)

FE Port 2 : Up, STP State : FORWARDING (Connects to - 0/FC0)

FE Port 3 : Up, STP State : FORWARDING (Connects to - 0/FC1)

FE Port 4 : Up, STP State : FORWARDING (Connects to - 0/AM0)

FE Port 5 : Up, STP State : FORWARDING (Connects to - 0/AM1)

FE Port 6 : Down (Connects to - )

FE Port 7 : Down (Connects to - )

FE Port 8 : Up, STP State : FORWARDING (Connects to - 0/SM0)

FE Port 9 : Up, STP State : FORWARDING (Connects to - 0/SM1)

FE Port 10 : Up, STP State : FORWARDING (Connects to - 0/SM2)

FE Port 11 : Up, STP State : FORWARDING (Connects to - 0/SM3)

FE Port 12 : Up, STP State : FORWARDING (Connects to - 0/SM4)

FE Port 13 : Up, STP State : FORWARDING (Connects to - 0/SM5)

FE Port 14 : Up, STP State : FORWARDING (Connects to - 0/SM6)

FE Port 15 : Up, STP State : FORWARDING (Connects to - 0/SM7)

GE Port 0 : Up, STP State : FORWARDING (Connects to - GE_0)

GE Port 1 : Up, STP State : FORWARDING (Connects to - Switch 1)

CRS Back to Back:-

Apart from CRS single chassis and CRS multi chassis we have one more variant available which is known as CRS back to back where two 16/8 slot CRS chassis are connected back to back without even using the Fabric card chassis ( FCC ). A back-to-back system is also called a 2+0 system because it includes two LCCs and zero FCCs. A CRS back-to-back multi chassis system provides the same capacity and capabilities as a traditional 2+1 multi chassis deployment. Pairing two LCCs to create a CRS back-to-back system does not require any dedicated slots for interconnection; switch fabric cards and optical cables connect the chassis to form a single logical system that maintains full bandwidth between chassis.

In CRS Back-to-Back System, fabric planes are divided into two stages: S1 and S3. The S2 stage is no longer needed. The purpose of the S2 stage is to direct traffic to the correct egress 16 slot chassis when there are multiple egress 16 slot chassis. In the CRS Back-to-Back System, there is only one egress 16 slot chassis.

Data arrives at the S1 stage in the ingress 8-slot chassis and then passes over the fabric cables to the S3 stage in the egress 8-slot chassis.

Below mentioned figure shows the simplified view of CRS back to back Connectivity.

CRS Back-to-Back.jpg

Verifying the Switch fabric operations:-

We have some commands that helps confirm if switch fabric is working fine or there is some issue.


RP/0/RP0/CPU0:Labcrs(admin)#show controllers fabric plane all detail

Flags: P - plane admin down, p - plane oper down

C - card admin down, c - card oper down

A - asic admin down, a - asic oper down

L - link port admin down, l - linkport oper down

B - bundle port admin Down, b - bundle port oper down

I - bundle admin down, i - bundle oper down

N - node admin down, n - node down

X - ctrl admin down, x - ctrl down

o - other end of link down d - data down

f - failed component downstream

m - plane multicast down, s - link port permanently shutdown

t - no barrier input O - Out-Of-Service oper down

T - topology mismatch down e - link port control only

D - plane admin data down

u - untunable g - tuning in progress

v - successfully tuned at least once

w - most recent tuning attempt failed

h - tuning pending z - rx-eye measurement in progress

Plane Admin Oper up->dn up->mcast Down Total Down

Id State State counter counter Flags Bundles Bundles

-------------------------------------------------------------------------

0 UP UP 0 1 27 18

1 UP UP 0 1 27 18

2 UP UP 0 1 27 18

3 UP UP 0 1 27 18

4 UP UP 1 2 27 18

5 UP UP 0 0 27 18

6 UP UP 0 0 27 18

7 UP UP 0 1 27 18

This command helps to if fabric planes are up or down and counters show how many times fabric planes has completely went down or multicast gone down ( e.g one of the fabric card in the plane goes down).

RP/0/RP0/CPU0:Labcrs(admin)#show controllers fabric connectivity all detail

Flags: P - plane admin down, p - plane oper down

C - card admin down, c - card oper down

A - asic admin down, a - asic oper down

L - link port admin down, l - linkport oper down

B - bundle port admin Down, b - bundle port oper down

I - bundle admin down, i - bundle oper down

N - node admin down, n - node down

X - ctrl admin down, x - ctrl down

o - other end of link down d - data down

f - failed component downstream

m - plane multicast down, s - link port permanently shutdown

t - no barrier input O - Out-Of-Service oper down

T - topology mismatch down e - link port control only

D - plane admin data down

u - untunable g - tuning in progress

v - successfully tuned at least once

w - most recent tuning attempt failed

h - tuning pending z - rx-eye measurement in progress

To Fabric From Fabric

Card In Tx Planes Rx Planes Monitored Total Percent

R/S/M Use 01234567 01234567 For (s) Uptime (s) Uptime

-------------------------------------------------------------------------------

0/0/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/3/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/4/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/5/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/7/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/8/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/9/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/RP0/CPU0 1 11111111 11111111 37363613 37363613 100.0000

0/RP1/CPU0 1 11111111 11111111 37363613 37363613 100.0000

1/0/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/1/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/2/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/3/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/4/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/5/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/6/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/8/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/9/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/10/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/11/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/12/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/13/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/14/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/15/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/RP0/CPU0 1 11111111 11111111 37363596 37363596 100.0000

1/RP1/CPU0 1 11111111 11111111 37363596 37363596 100.0000

This command shows the connectivity details for all the line cards and RP’s against the fabric planes.

Some other useful commands are.

  • Show controllers fabric link health
  • Show controllers fabric plane all statistics
  • Show controllers fabric bundle summary
  • Show controllers fabric trace.

In case you face some fabric related issue with CRS router above mentioned commands outputs helps In identifying the reason for failures.

Watch the Tech-Talk and you may also download the Presentation


3 Comments
Manish Kumar
Cisco Employee

This is very good article, thanks for sharing..

Manish

vinayk3
Cisco Employee

Thanks Manish.

Vinay,Rahul & Hitesh

Rishi_Bhardwaj
Beginner

Dear Team,

Can you please explain Netflow, CEF and Openflow mechanisms in brief. I found that netflow and CEF can be enabled on an interface simultaneously. How both technologies compliment each other and how openflow is different from these and works along with them?

Thanks

Rishi Bhardwaj

Content for Community-Ad

This widget could not be displayed.