Switch Fabric Capacity Average Utilization Peak Utilization on ASR 9010 using XOR 5.3.3

[Bilal Raziq]

Is there any command on XOR to check Switch Fabric Average/Peak utilization .We have single RSP 440 which has switch fabric capacity of 220 Gbps . Trying to find how much more traffic can be added to ASR .

Regards

Bilal

[Aleksandar Vidakovic · Accepted solution]

hi Bilal,

these commands will show you the fabric utilisation info:

sh controllers fabric fia stats location <location>
sh controllers fabric fia q-depth location <location>
sh controllers fabric fia drops ingress location <location>
sh controllers fabric fia drops egress location <location>

There is no rate counters per se, but by checking the drops and q-depth you will see when the limits are reached.

Also look up the BRKSPG-2904 from Cisco Live Milan 2014. There is a good section on fabric throubleshooting.

Hope this helps,

/Aleksandar

[Bilal Raziq]

Thanks Aleksandar

The drops are zero ...

But i need to plan to put more traffic on the machine

Can you help me understand the output of two commands

RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia q-depth location 0/RSP1/CPU0
Wed Mar 1 12:54:42.625 QAT

********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri pktcnt Slot_FIA_NP

********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri pktcnt Slot_FIA_NP
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia errors location 0/RSP1/CPU0
^
% Invalid input detected at '^' marker.
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia error
% Incomplete command.
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia errors
% Incomplete command.
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia errors ?
egress Egress Direction
ingress Ingress Direction
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia errors egress
Wed Mar 1 12:55:26.439 QAT

********** FIA-0 **********
Category: eg_error-0
To Spaui Error-0 0
To Spaui Error-1 0
RL over/under flow cnt 0
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia errors ingress
Wed Mar 1 12:55:40.734 QAT

********** FIA-0 **********
Category: in_error-0
To Xbar Uc Crc-0 0
To Xbar Uc Crc-1 0
To Xbar Uc Crc-2 0
To Xbar Uc Crc-3 0
To Xbar Mc Crc-0 0
To Xbar Mc Crc-1 0
To Xbar Mc Crc-2 0
To Xbar Mc Crc-3 0
nb pa read data err 0
pa header err 0
pa crc16 err 0
pa crc32 err 0
pa to tf err 0
ab overflow req lost 0
ni bad crc32 0
ni crc32 corrupt 0
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia stats location 0/RSP1/CPU0
Wed Mar 1 12:56:13.509 QAT

********** FIA-0 **********
Category: count-0
From Unicast Xbar[0] 1488209889
From Unicast Xbar[1] 0
From Unicast Xbar[2] 2385264406
From Unicast Xbar[3] 0
From MultiCast Xbar[0] 240128
From MultiCast Xbar[1] 0
From MultiCast Xbar[2] 0
From MultiCast Xbar[3] 0
To Unicast Xbar[0] 1922366536
To Unicast Xbar[1] 0
To Unicast Xbar[2] 0
To Unicast Xbar[3] 0
To MultiCast Xbar[0] 240135
To MultiCast Xbar[1] 0
To MultiCast Xbar[2] 0
To MultiCast Xbar[3] 0
To Line Interface[0] 3874049220
To Line Interface[1] 176
From Line Interface[0] 1922602945
From Line Interface[1] 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia stats location 0/RSP1/CPU0
Wed Mar 1 12:56:42.853 QAT

********** FIA-0 **********
Category: count-0
From Unicast Xbar[0] 1488216041
From Unicast Xbar[1] 0
From Unicast Xbar[2] 2385274036
From Unicast Xbar[3] 0
From MultiCast Xbar[0] 240130
From MultiCast Xbar[1] 0
From MultiCast Xbar[2] 0
From MultiCast Xbar[3] 0
To Unicast Xbar[0] 1922374451
To Unicast Xbar[1] 0
To Unicast Xbar[2] 0
To Unicast Xbar[3] 0
To MultiCast Xbar[0] 240137
To MultiCast Xbar[1] 0
To MultiCast Xbar[2] 0
To MultiCast Xbar[3] 0
To Line Interface[0] 3874065004
To Line Interface[1] 176
From Line Interface[0] 1922610862
From Line Interface[1] 0
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RSP1/CPU0:LTE_APC_ASR01#
RP/0/RSP1/CPU0:LTE_APC_ASR01#
RP/0/RSP1/CPU0:LTE_APC_ASR01#
RP/0/RSP1/CPU0:LTE_APC_ASR01#sh controllers fabric fia stats location 0/RSP1/CPU0
Wed Mar 1 12:59:44.338 QAT

********** FIA-0 **********
Category: count-0
From Unicast Xbar[0] 1488252423
From Unicast Xbar[1] 0
From Unicast Xbar[2] 2385332970
From Unicast Xbar[3] 0
From MultiCast Xbar[0] 240136
From MultiCast Xbar[1] 0
From MultiCast Xbar[2] 0
From MultiCast Xbar[3] 0
To Unicast Xbar[0] 1922422205
To Unicast Xbar[1] 0
To Unicast Xbar[2] 0
To Unicast Xbar[3] 0
To MultiCast Xbar[0] 240143
To MultiCast Xbar[1] 0
To MultiCast Xbar[2] 0
To MultiCast Xbar[3] 0
To Line Interface[0] 3874160351
To Line Interface[1] 176
From Line Interface[0] 1922658622
From Line Interface[1] 0
Ingress drop: 0
Egress drop: 0
Total drop: 0

[Aleksandar Vidakovic · Accepted solution]

hi Bilal,

the q-depth command shows you how many packets are sitting in the ingress FIA queue, waiting to be transmitted towards the egress FIA+NP. If you see packets there when you run the command, it means there is back-pressure from the egress side. Back-pressure is explained well in the BRKARC-2003 or BRKSPG-2904 from Cisco Live Berlin 2016.

The "fia errors" command shows you any layer 1 issues. You don't need to run this unless you suspect any low HW issue on the fabric path.

The "fia stats" and "fia drops" are explained in BRKSPG-2904 from Cisco Live Milan 2014. Did you manage to download the presentation?

regards,

/Aleksandar

[Adam Vitkovsky]

Hi Aleksandar,

With the advent of NPU telemetry, we’ll we have a command displaying the CPU utilization on each of the PPEs on the NPU or the NPU as a whole please?

Looking at the FIA stats of ingress LCs is somewhat indirect measurement of a particular egress NPU utilization.

What if the NPU is overloaded by traffic coming in from a WAN side (not Fabric side), the one might want to check for any EFD drops.

So I’m just thinking whether monitoring the ICFDQs wouldn’t be a better approach to assess the NPU status please? Since packet heads from both WAN and Fabric will be stored in these queues, then if one sees them filling up or experiencing drops (the lower priority queues) one can infer that the NPU is becoming overloaded right please?

adam

netconsultings.com

::carrier-class solutions for the telecommunications industry::

[Ashish Panda]

Hi Adam,

EFD drops can be due to congestion on any of the TOP tasks in the NP pipeline resulting in flow control and EFD drops. This can indicate NP getting overwhelmed with ingress/WAN traffic.

The EFD drops can be tracked with the following

show controllers np fast-drop np<> location 0/<>/CPU0

There are internal (TAC/BU) shell commands to check the NP load / performance as well.

regards

Ashish

[Adam Vitkovsky]

Can you please let me know the shell commands to check the NP load / performance

-I’d like to be able to monitor what headroom I have for enabling features or for introducing more pps load to the NPU.

Yes please,

I understand that EDF drops = overload from WAN.

EFD takes care only about ICFDQs used for WAN input (the packet heads in them) – But I actually don’t know how many of the 64 groups x 4 CoS queues are dedicated to WAN input.

(the need for EFD in the “from-WAN” direction is because there’s no backpressure mechanism in this direction that the NPU can use, well other than Ethernet flow control if enabled or supported by the MPA).

But my understanding is,

That some of these ICFDQs (some of the 64 groups x 4 CoS queues) are used to hold packet heads of frames coming from FIA (Fabric-side) that are waiting to be processed by the NPU.

-so by monitoring the fullness of these ICFDQs (feeder queues) one could infer whether the NPU is becoming clogged (from WAN of from Fabric or both).

-one thing I just realized I’m actually not sure about is whether:

A) NPU initiates backpressure towards FIA (ordering FIA to throttle fabric grants) based on fullness of these NPU feeder queues or based on NPU utilization.

Or

B) FIA knows when it has to initiate backpressure because it monitors fullness of these NPU feeder queues.

Can you please clarify?

Thank you very much

adam

netconsultings.com

::carrier-class solutions for the telecommunications industry::

[Aleksandar Vidakovic]

hi Adam,

you can use the "np_perf -e <n>" shell command to see the summary of NP performance info. Most of this output is for developers' consumption, but you will find a section that specifies the PPS rate, TOP utilisation in % and current RFD occupation.

Look up BRKSPG-2904 from Cisco Live Berlin 2016 or 2017. There's a section on "Tomahawk vs Typhoon EFD" that explains the back-pressure mechanism from NP to FIA.

/Aleksandar

[Adam Vitkovsky]

Geez I’m stupid sorry to bother you it’s all in my notes from our previous conversation, it’s just the doc has 50 pages :)

But search for RFD reveals the answer I was looking for,

But what I’m not sure about is the relationship between ICFDQs and RFD queues? Is it WAN vs Fabric?

Thank you very much for the cmd

Would you consider adding output of this cmd to streaming telemetry please?

I’d like to look at the network weather-map and see, "aah these NPUs are in amber region, so we first need to migrate some stuff to the green NPUs, before rolling out netflow and urpf on these ports".

Or talking SDN –having my network controller provision services and their paths using also this information.

adam

netconsultings.com

::carrier-class solutions for the telecommunications industry::