ASR9K Chassis Fabric Modes

[saurabja]

Introduction

This document explains the use of different fabric modes on ASR9912, ASR9910, ASR9922 chassis models of ASR9000 platform.

This document is not applicable for ASR9001, ASR9010, ASR9006, ASR9904 chassis. These chassis types only need the default fabric mode without any explicit admin configuration.

32-bit IOS XR

Default Mode

Supported LC/RP/FC:

LC	Typhoon, Tomahawk
RP	ASR-9900-RP, A99-RP2
FC	ASR-9900-SFC, A99-SFC2

In default mode Typhoon and Tomahawk LCs can be inter-mixed in the chassis. The number of VQIs are limited to 1024 and multicast traffic uses only first 5 fabric cards. No explicit admin configuration is required to enable this mode.

HighBandWidth Mode

Supported LC/RP/FC:

LC	Tomahawk
RP	A99-RP2
FC	A99-SFC2

The number of VQIs are up to 2048 and multicast traffic uses only first 5 Fabric cards. Both Tomahawk 5-fabric (A9K- LC PID) and 7-fabric (A99- LC PID) LCs can be used in the chassis. Typhoon LCs are not supported in this mode. It is recommended that chassis has all 7 fabric cards. This mode is enabled by using the following admin config CLI:

fabric enable mode highbandwidth

This CLI would be rejected if the chassis has an unsupported card that should be removed prior to doing a config commit.

A99-HighBandWidth Mode

Supported LC/RP/FC:

LC	A99 (7-fabric) Tomahawk
RP	A99-RP2
FC	A99-SFC2

The number of VQIs are up to 2048 and multicast traffic uses all 7 fabric cards. This mode is enabled by using the following admin config CLI:

fabric enable mode A99-highbandwidth

This CLI would be rejected if the chassis has an unsupported card that should be removed prior to doing a config commit.

Switching from default to HighBandWidth mode doesn't require router reload. Similarly, switching from default or HighBandWidth to A99-HighBandWidth mode doesn't require router reload. However, switching from any other mode to default requires a router reload. After selecting the chassis mode, if an unsupported card is inserted in the chassis it may not cleanly boot up and may ultimately move to IN-RESET state.

In default and HighBandWidth mode, the egress FIA rate limiter (enabled upon fabric degrade) calculation is based on status of first 5 fabric cards. However, in A99-HighBandWidth mode, the egress FIA rate limiter calculation is based on status of all 7 fabric cards.

64-bit IOS XR

Note: Typhoon line cards are not supported in 64-bit IOS XR

Default Mode

Supported LC/RP/FC:

LC	Tomahawk, Lightspeed
RP	A99-RP2 A99-RP3
FC	A99-SFC2 A99-SFC3

The number of VQIs are up to 2048 and multicast traffic uses only first 5 fabric cards. Both the 5-fabric (A9K- LC PID) and 7-fabric (A99- LC PID) LCs can be used in the chassis. It is recommended that chassis has all 7 fabric cards.

Two important points related to use of a 7-fabric 32x100G Lightspeed line card in this mode:

• A99-SFC3 cards must be used in order to provide the required full BW to this card. It is recommended that all 7 switch fabric cards are installed.
• FIA rate limiter is applied in this fabric mode to police the traffic to 375Gbps per slice. When all four ports are enabled, the 375Gbps are shared between them. If three or less ports are enabled, each port can achieve the full 100Gbps. When the policer is applied this message is reported in the syslog:

%FABRIC-FIA-1-RATE_LIMITER_ON : Set|fialc[4320]|0x108a000|Insufficient fabric capacity for card types in use - FIA egress rate limiter applied

A99-HighBandWidth Mode

Supported LC/RP/FC:

LC	A99 (7-fabric) Tomahawk A99 (7-fabric) Lightspeed
RP	A99-RP2 A99-RP3
FC	A99-SFC2 A99-SFC3

The number of VQIs are up to 2048 and multicast traffic uses all 7 fabric cards. If a 7-fabric Lightspeed line card is in the chassis, A99-SFC3 cards must be used in order to provide the required full BW to this card.

[smailmilak]

Hi,

thanks for the document!

Is the MOD400-TR and MOD200-TR Tomahawk LC a 7-FAB LC?

We are using 7x SFCv2 and RP2 TR. All in 9912.

[Aleksandar Vidakovic]

A9K- line cards are 5-fabric

A99- line cards are 7-fabric

[smailmilak]

Thanks Aleks!

[dpothier]

Update: With the fix for CSCuz01748 a router reload is no longer required.  CSCuz01748 is fixed in XR version 6.0.2 and 6.1.1, and onward versions, so the router reload is not required. If you are using prior XR versions than 6.0.2 and 6.1.1 then a router reload is required. Example if you are running 5.3.2, then the router reload is still required.

CSCuz01748 Config change from HighBWMode to A99-HighBWMode should be non-reload

[Charlene Doherty]

Thanks for the document.

So to confirm my customer running 5.3.1 plans move to 'high bandwidth mode' from default mode with the command <fabric enable mode highbandwidth>

This will require a reload as per the comment from David below?

The plan is to upgrade to 6.1.x in the future.  However according to the 6.1.2 release notes The fabric enable mode highbandwidth command has been deprecated from Release 6.1.2.  http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r6-1/general/release/notes/b-release-note-asr9k-612.html 

What is the reason for this?

How can we enable 2048 queues without this command in later releases?

Thank You!

[Aleksandar Vidakovic]

Hi Charlene,

the "fabric enable mode highbandwidth command" was deprecated in 64-bit XR because on this flavour of XR we don't support Typhoon linecards. Hence the highbandwidth mode is default in 64-bit XR.

This command is still available on the 32-bit XR:

RP/0/RP1/CPU0:LSR1#sh install active summary
Default Profile:
  SDRs:
    Owner
  Active Packages:
    disk0:asr9k-mini-px-6.2.2.19I
    disk0:asr9k-mcast-px-6.2.2.19I
    disk0:asr9k-doc-px-6.2.2.19I
    disk0:asr9k-mgbl-px-6.2.2.19I
    disk0:asr9k-fpd-px-6.2.2.19I
    disk0:asr9k-mpls-px-6.2.2.19I
    disk0:asr9k-video-px-6.2.2.19I
    disk0:asr9k-k9sec-px-6.2.2.19I
    disk0:asr9k-li-px-6.2.2.19I
    disk0:asr9k-px-6.2.2.19I.CSCvd33154-0.0.2.i

RP/0/RP1/CPU0:LSR1#admin
RP/0/RP1/CPU0:LSR1(admin)#config
RP/0/RP1/CPU0:LSR1(admin-config)#fabric enable mode ?
  A99-highbandwidth  A99 High bandwidth cards only
  highbandwidth      High bandwidth cards only
RP/0/RP1/CPU0:LSR1(admin-config)#fabric enable mode

hth,

/Aleksandar

[Charlene Doherty]

Thanks for the clarification Aleks! 

[guferrar]

Great doc! 

I Have a doubt regarding ASR9910/9912 won´t need to be changed from default mode since they support a maximum of 10 slots and the LC A9K-8x100GE ( highest interface density over RSP-880)  requires 96 VQIs. 

is that correct?

best regards,

guferrar

[AARON WEINTRAUB]

What is the show command to display the current fabric mode?