How to monitor N7K frabic throughput

[fsebera]

Hi,

We have several Nexus 7010s running NX-OS 6.2 and have populated just enough fabric module slots to support the cards installed.

The cli show hardware fabric-utilization shows the real-time load across the fabrics.

Is there a method to show the historical values of the load across the fabric modules over the last month or 6 months?

Is a monitoring application required to capture and collect this throughput?

Thank you

Frank

[jeestrad]

Hello Frank,

Can you try the next command:

show hardware internal xbar-driver xbar <xbar> inst <inst> counters all util

Is that the information you need?

[fsebera]

This does provide additional insightful details but doesn't seem to provide historical views a period of time.

I think the assumption here is the line cards would never overload the fabric ...... even if all cards were running at line rate so monitoring the fabric is not really necessary - is this correct?

Thanks Frank

[Madhukrishnan Gopinathan Nair]

Hello Frank,

When you say Just enough, do you mean minimum xbars to bring up the modules?

Any time, it would be good to have 1 additional in terms of redundancy in failure condition of xbar module

Thanks,

Madhu

[fsebera]

Hi Madhu,

YES and this is just what I am trying to figure out.

We currently have 3 fabric modules installed. Are we at capacity with these 3?  I don't know and having a tough time trying to figure out what the current load is.

ANY help would be GREATLY APPRECIATED!!!!!

Does Cisco have ANY recommendations on this?

Thank you

Frank

[Madhukrishnan Gopinathan Nair · Accepted solution]

Hello Frank,

If you follow the naming convention of the module, for eg. N7K-F248XP-25E  needs to have 2 Fabric modules to bring it up and for its max capacity, it needs to have 5 Fabric modules. 

However it really depends on how much traffic is flowing around this box whether you need to all of them. You could use the below commands to figure out if you crossing or nearing the capacity of the system

show hardware fabric-utilization
show hardware fabric-utilization detail

The below command shows you the peak utilization of the fabric also

show hardware capacity fabric-utilization

If you are seeing high utilization, may be you could consider adding more fabric.

Hope this helps and rate the posts if found to be useful.

Thanks,

Madhu

[fsebera]

Hi Madhu,

THANK YOU!!

The last command your provided did the trick!

It appears our N7k is barely processing traffic as 9% util is the max as shown below.

Each VDC provides the same date & time utilization stats so assuming this is a global command.

Thank you again

Frank

NX7K_A# sh hardware capacity fabric-utilization

------------------------------------------------------------------------------
Fabric Planes:
A -- Unicast fabric packets
B -- Multicast/Multidestination fabric packets
------------------------------------------------------------------------------
Bandwidth is in Gbps and shared by both Fabric Planes (A+B)
----------------------------PEAK FABRIC UTILIZATION-----------------------------
Mod Fab Fab Fab  ASIC Band Fab      Ingress                  Egress
    Lnk Mod ASIC Port wdth Pln Util%       Time         Util%       Time
--------------------------------------------------------------------------------
1    1   1    1    5   23   A     4 16-02-14@21:56:03    0         --
1    1   1    1    5   23   B     4 16-01-24@22:04:33    4 16-01-24@22:04:33
1    2   1    1    3   23   A     4 16-02-14@21:56:03    0         --
1    2   1    1    3   23   B     0       --             1 16-01-24@23:37:24
1    3   2    1    5   23   A     4 16-02-14@22:09:45    1 16-01-04@14:56:08
1    3   2    1    5   23   B     0       --             0         --
1    4   2    1    3   23   A     4 16-02-14@21:56:03    1 16-01-11@20:04:49
1    4   2    1    3   23   B     0       --             4 16-01-25@01:10:15
1    5   3    1    5   23   A     4 16-02-14@22:09:45    0         --
1    5   3    1    5   23   B     0       --             4 16-01-24@22:04:33
1    6   3    1    3   23   A     4 16-02-14@21:52:43    0         --
1    6   3    1    3   23   B     0       --             0         --
5    1   1    1   11   23   A     0       --             0         --
5    1   1    1   11   23   B     0       --             0         --
5    2   2    1   11   23   A     0       --             0         --
5    2   2    1   11   23   B     0       --             0         --
5    3   3    1   11   23   A     0       --             0         --
5    3   3    1   11   23   B     0       --             0         --
6    1   1    1    7   23   A     0       --             0         --
6    1   1    1    7   23   B     0       --             0         --
6    2   2    1    7   23   A     0       --             0         --
6    2   2    1    7   23   B     0       --             0         --
6    3   3    1    7   23   A     0       --             0         --
6    3   3    1    7   23   B     0       --             0         --
7    1   1    1   15   23   A     0       --             0         --
7    1   1    1   15   23   B     0       --             4 16-01-24@22:00:28
7    2   1    1    1   23   A     0       --             0         --
7    2   1    1    1   23   B     0       --             3 16-01-24@23:37:20
7    3   2    1   15   23   A     0       --             0         --
7    3   2    1   15   23   B     0       --             0         --
7    4   2    1    1   23   A     0       --             0         --
7    4   2    1    1   23   B     0       --             4 16-01-24@23:21:58
7    5   3    1   15   23   A     0       --             0         --
7    5   3    1   15   23   B     0       --             3 16-01-24@22:00:28
7    6   3    1    1   23   A     0       --             0         --
7    6   3    1    1   23   B     0       --             0         --
8    1   1    1   10   23   A     9 16-01-27@17:15:29    9 16-01-27@16:08:01
8    1   1    1   10   23   B     0       --             4 16-01-24@23:09:27
8    2   1    1   16   23   A     9 16-01-27@17:15:29    9 16-01-27@16:08:01
8    2   1    1   16   23   B     2 16-01-24@23:36:10    2 16-01-24@23:36:10
8    3   2    1   10   23   A     9 16-01-27@16:20:43    8 16-02-03@10:36:13
8    3   2    1   10   23   B     0       --             0         --
8    4   2    1   16   23   A     9 16-01-27@16:20:43    8 16-02-03@10:36:13
8    4   2    1   16   23   B     4 16-01-25@00:50:59    4 16-01-25@00:50:59
8    5   3    1   10   23   A     8 16-01-28@08:49:03    8 16-01-28@08:49:03
8    5   3    1   10   23   B     4 16-01-24@20:38:48    4 16-01-24@20:38:48
8    6   3    1   16   23   A     8 16-01-27@17:17:29    8 16-01-27@16:56:47
8    6   3    1   16   23   B     0       --             0         --
10   1   1    1    0   23   A     2 16-02-08@02:35:50    2 16-02-09@00:07:06
10   1   1    1    0   23   B     0         --             4 16-01-24@20:45:11
10   2   1    1    8   23   A     2 16-02-08@02:35:50    2 16-02-09@00:07:06
10   2   1    1    8   23   B     0       --             0         --
10   3   2    1    0   23   A     2 16-02-13@02:18:37    2 16-02-09@19:35:27
10   3   2    1    0   23   B     0       --             0         --
10   4   2    1    8   23   A     2 16-02-13@02:18:37    2 16-02-09@19:35:27
10   4   2    1    8   23   B     1 16-01-21@15:06:43    4 16-01-25@00:51:01
10   5   3    1    0   23   A     2 16-02-08@02:42:51    2 16-02-09@19:43:48
10   5   3    1    0   23   B     0       --             5 16-01-24@20:37:50
10   6   3    1    8   23   A     2 16-02-08@02:42:51    2 16-02-09@19:43:48
10   6   3    1    8   23   B     0       --             0         --

[Madhukrishnan Gopinathan Nair]

Thank you and happy to see it helped you Frank. Could you please mark this question as answered also, so that it will be useful for others as well.

Thanks,
Madhu.