04-27-2016 07:08 AM - edited 03-01-2019 12:42 PM
We recently upgraded our UCS Cluster from 2.2(3f) to 2.2(6e).
I didn't use the auto upgrade, i used set of directions provided to me by TAC.
Since the upgrade, Both of my IOM (2208XP's) throw the following errors:
sys/chassis-1/slot-2/fabric/port-4/fault-F43324
TCA: etherNiErrStats crcDelta, current-value = 54, raised above esc value 5
I've seen it on Slot2/Port4, Slot1/Port1, and Slot1/Port2.
Thanks
Solved! Go to Solution.
04-29-2016 07:52 AM
Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.
Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|
N5K bug: CSCuj86736
UCS bug: CSCuo76425
The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.
Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.
04-27-2016 07:26 AM
Hey Michael,
These errors are reporting that there is are CRC errors being logged on your IOM interfaces. The easiest way to try and narrow this down is swapping the cable/optics on those interfaces and monitor to see if they are still being generated.
I believe in the 2.2.6 train we introduced different thresholds for errors in that a fault would be generated once we reach or cross a certain value. If you are unable to isolate the issue, you may opt to open a TAC case to have them investigate further.
Let me know if you need further clarification.
Wes
04-28-2016 08:24 AM
Michael,
UCSM 2.2.4 adds the following feature to monitor IOM NIF port errors and by default triggers a warning if the number of errors increase the value of 5. You can use this feature to automatically shut down the port if the CRCs go beyond a particular value.
http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html#pgfId-282711
Policy-Based Port Error Handling—If Cisco UCS Manager detects any errors on active IOM Network Interface (NIF) ports, and if the Error-disable setting is enabled, Cisco UCS Manager automatically disables the respective FI port that is connected to the IOM NIF port that had errors.
When a FI port is error disabled, it is effectively shut down and no traffic is sent or received on that port.
The config can be checked/modified as shown in link below
04-28-2016 10:23 AM
Hi [@me@mpking.com],
Can you SSH to UCS and send the output of the commands below?
UCS-A# connect nxos b
UCS-B(nxos)# show hardware internal carmel crc
UCS-B(nxos)# show hardware internal carmel eye
UCS-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
04-28-2016 12:16 PM
I don't think I can get this to display right
Interop-A# connect nxos b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Interop-B(nxos)# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 68 | 68 |
| Eth 1/2 | --- | --- | --- | --- | --- | 58 | 58 |
| Eth 1/3 | --- | --- | --- | --- | --- | 1748 | 1748 |
| Eth 1/4 | --- | --- | --- | --- | --- | 378933 | 378933 |
| Eth 1/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | 252571 | --- | 252571 | --- | --- | --- |
| Eth 1/20 | --- | 129712 | --- | 129710 | --- | --- | --- |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/29 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/30 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/34 | --- | --- | --- | --- | --- | 1648 | 1648 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
Interop-B(nxos)# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 171 mv | 828 mUI | 37/ 35 | 04/28/2016 15:08:37.487235 |a9|ec|e2|2d|90|6c|e2|2d|88|00|
fi0 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:37.492363 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:37.497490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 153 mv | 828 mUI | 31/ 35 | 04/28/2016 15:08:43.594756 |a9|ec|4a|2f|b0|6c|4a|2f|88|00|
fi0 | 200 mv | 828 mUI | 40/ 35 | 04/28/2016 15:08:43.599885 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.605015 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 162 mv | 812 mUI | 34/ 34 | 04/28/2016 15:08:49.707225 |a9|ec|2c|2d|b0|6c|2c|2d|88|00|
fi0 | 200 mv | 921 mUI | 40/ 3b | 04/28/2016 15:08:49.712355 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:49.717481 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 134 mv | 718 mUI | 2b/ 2e | 04/28/2016 15:08:52.787242 |a9|ea|2c|0b|b0|6a|2c|0b|88|00|
fi0 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:52.792370 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:52.797498 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:15.998076 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.028490 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.071352 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.101230 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.131251 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.149960 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.191241 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.209556 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.270749 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.298956 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.324681 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.346728 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 118 mv | 750 mUI | 26/ 30 | 04/28/2016 15:08:37.502645 |a9|dd|9c|25|b0|5d|9c|25|88|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:37.507769 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:37.512900 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 112 mv | 734 mUI | 24/ 2f | 04/28/2016 15:08:43.610162 |a9|da|86|43|92|5a|86|43|88|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:43.615292 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.620417 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 121 mv | 671 mUI | 27/ 2b | 04/28/2016 15:08:49.722629 |a9|d4|a2|67|b0|54|a2|67|88|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:49.727756 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:49.732885 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 121 mv | 796 mUI | 27/ 33 | 04/28/2016 15:08:52.802745 |a9|de|c6|23|f0|5e|c6|23|88|00|
fi0 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:52.807880 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:52.813009 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.505055 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.538961 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.553591 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.574806 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.598739 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.619035 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.644852 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.678867 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.693460 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.719028 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Interop-B(nxos)#
Interop-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
Slot 0 Carmel 2 interrupt statistics:
car_bm3_INT_err_ig_mtu_vio |3 |0 |3 |0
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Interop-B(nxos)#
04-28-2016 05:48 PM
It looks like you're receiving bad frames from your upstream switches (Eth1/19, Eth1/20) and these are getting stomped and being sent to the IOM (Eth1/1 - Eth1/4).
You need to find out where these are coming from. What kind of switches do you have upstream? If they are Nexus 5548/5596 you can run the same first two commands I sent.
04-29-2016 06:15 AM
It's a Nexus 5548UP (5.1(3)N2(1c)). Those ports goto the 2nd Nexus. Ports 11/12
9/10 are a channel group to FI-A (channel group 50)
11/12 are a channel group to FI-B (channel group 51)
There is a VPC-Peer-link on 1/29 and 1/30 (PO1)
and downstream switch on 1/28 (PO2)
IO-Nexus-02# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 403697 | 403697 |
| Eth 1/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/5 | --- | --- | --- | --- | --- | 3056 | 3056 |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | 130434 | --- | --- | 130087 | --- | 1387792 | 1387792 |
| Eth 1/10 | --- | --- | --- | --- | --- | 189038 | 189038 |
| Eth 1/11 | 2537118 | --- | --- | 2532078 | --- | 294708 | 294708 |
| Eth 1/12 | --- | --- | --- | --- | --- | 147652 | 147652 |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | --- | --- | --- | --- | 3144 | 3144 |
| Eth 1/20 | --- | --- | --- | --- | --- | 21917 | 21917 |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | 138 | 138 |
| Eth 1/29 | --- | --- | --- | --- | --- | 3641 | 3641 |
| Eth 1/30 | --- | --- | --- | --- | --- | 96995 | 96995 |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | 586 | 586 |
| Eth 1/34 | --- | --- | --- | --- | --- | 1004 | 1004 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 2/1 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/15 | --- | --- | --- | 1 | --- | --- | --- |
| Eth 2/16 | --- | --- | --- | 1 | --- | --- | --- |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 3/1 | 51746 | --- | --- | 51741 | --- | 16016 | 16016 |
| Eth 3/2 | 51296 | --- | --- | 51282 | --- | 41508 | 41508 |
| Eth 3/3 | 57716 | --- | --- | 57706 | --- | 19237 | 19237 |
| Eth 3/4 | 57536 | --- | --- | 57527 | --- | 10478 | 10478 |
| Eth 3/5 | 61480 | --- | --- | 61471 | --- | 16712 | 16712 |
| Eth 3/6 | 55210 | --- | --- | 55200 | --- | 176914 | 176914 |
| Eth 3/7 | 54444 | --- | --- | 54435 | --- | 163659 | 163659 |
| Eth 3/8 | 60945 | --- | --- | 60935 | --- | 15425 | 15425 |
| Eth 3/9 | 54553 | --- | --- | 54544 | --- | 18428 | 18428 |
| Eth 3/10 | 54639 | --- | --- | 54628 | --- | 24437 | 24437 |
| Eth 3/11 | 2 | --- | --- | 2 | --- | 12940 | 12940 |
| Eth 3/12 | 60311 | --- | --- | 60301 | --- | 8095 | 8095 |
| Eth 3/13 | 56249 | --- | --- | 56239 | --- | 304267 | 304267 |
| Eth 3/14 | 58109 | --- | --- | 58104 | --- | 13854 | 13854 |
| Eth 3/15 | 59819 | --- | --- | 59812 | --- | 27124 | 27124 |
| Eth 3/16 | 49437 | --- | --- | 49432 | --- | 153811 | 153811 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.388864 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.404816 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.420032 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.445690 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 121 mv | 734 mUI | 27/ 2f | 04/29/2016 09:10:00.858343 |a9|e3|a2|01|90|63|a2|01|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.863470 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:10:00.868589 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.475343 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.487316 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.500157 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.471003 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.476130 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 121 mv | 781 mUI | 27/ 32 | 04/29/2016 09:09:45.568212 |a9|da|6e|6a|f0|5a|6e|6a|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:45.573328 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/29/2016 09:09:45.578447 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:09:51.673940 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 828 mUI | 40/ 35 | 04/29/2016 09:09:51.679068 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 121 mv | 718 mUI | 27/ 2e | 04/29/2016 09:09:57.768373 |a9|d0|6c|28|f0|50|6c|28|88|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:57.773503 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.778641 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.560631 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.572713 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.584407 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.596294 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.611372 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.626295 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 59 mv | 562 mUI | 13/ 24 | 04/29/2016 09:09:51.684219 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:51.689357 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 812 mUI | 40/ 34 | 04/29/2016 09:09:51.694490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 103 mv | 656 mUI | 21/ 2a | 04/29/2016 09:09:57.784242 |a9|de|c0|00|10|5e|c0|00|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:57.789384 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.794516 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.669531 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.684379 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.700335 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.713472 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.726645 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.739353 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.752351 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:42:57.176627 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 131 mv | 750 mUI | 2a/ 30 | 04/29/2016 09:10:00.873831 |a9|e6|ce|19|b2|66|ce|19|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.878952 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:10:00.884073 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 128 mv | 750 mUI | 29/ 30 | 04/29/2016 09:09:21.145872 |a9|df|88|23|d0|5f|88|23|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:21.151004 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:21.156141 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.862569 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.903761 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/2 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/4 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/3 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/6 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/5 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/8 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/7 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/9 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/10| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/11| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/12| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/13| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/14| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/15| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/16| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.257487 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.261558 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.265687 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.270909 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.276357 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.280462 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.284353 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.292033 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.298433 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.303185 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.309944 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.316020 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.320194 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.243841 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.247526 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.253393 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
car_bm0_INT_err_ig_mtu_vio |1 |0 |1 |0
car_bm2_INT_err_ig_mtu_vio |194 |0 |4 |0
Slot 0 Carmel 2 interrupt statistics:
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Slot 1 Carmel 0 interrupt statistics:
Slot 1 Carmel 1 interrupt statistics:
Slot 2 Carmel 0 interrupt statistics:
Slot 2 Carmel 1 interrupt statistics:
04-29-2016 07:52 AM
Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.
Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|
N5K bug: CSCuj86736
UCS bug: CSCuo76425
The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.
Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.
05-02-2016 06:09 PM
[@me@mpking.com] , did the above resolve your issue?
05-04-2016 06:27 AM
Yes, this appears to have been it.
I see this bug has been patched, so I will schedule a maintenance window to move up.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide