cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

9132
Views
0
Helpful
9
Replies
Beginner

TCA: etherNiErrStats crcDelta

We recently upgraded our UCS Cluster from 2.2(3f) to 2.2(6e).

I didn't use the auto upgrade, i used set of directions provided to me by TAC.

Since the upgrade, Both of my IOM (2208XP's) throw the following errors:

sys/chassis-1/slot-2/fabric/port-4/fault-F43324

TCA: etherNiErrStats crcDelta, current-value = 54, raised above esc value 5

I've seen it on Slot2/Port4, Slot1/Port1, and Slot1/Port2.

  • Is this a known bug on 2.2(6e)? I can't find anything after searching.
  • Is it new behavior (I didn't see anything in the release notes)
    • Could it be I've always had a problem, and UCS is just now alerting on it?
  • Did the firmware upgrade go sideways, and now I have a ticking time bomb?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

Eth1/19 and Eth1/11 are the

Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.

Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|

Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|

N5K bug: CSCuj86736

UCS bug: CSCuo76425

The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.

Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.

9 REPLIES 9
Cisco Employee

Hey Michael,

Hey Michael,

These errors are reporting that there is are CRC errors being logged on your IOM interfaces. The easiest way to try and narrow this down is swapping the cable/optics on those interfaces and monitor to see if they are still being generated.

I believe in the 2.2.6 train we introduced different thresholds for errors in that a fault would be generated once we reach or cross a certain value. If you are unable to isolate the issue, you may opt to open a TAC case to have them investigate further.

Let me know if you need further clarification. 

Wes

Highlighted
Cisco Employee

Michael,

Michael,

UCSM 2.2.4 adds the following feature to monitor IOM NIF port errors and by default triggers a warning if the number of errors increase the value of 5. You can use this feature to automatically shut down the port if the CRCs go beyond a particular value.

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html#pgfId-282711

Policy-Based Port Error Handling—If Cisco UCS Manager detects any errors on active IOM Network Interface (NIF) ports, and if the Error-disable setting is enabled, Cisco UCS Manager automatically disables the respective FI port that is connected to the IOM NIF port that had errors.

When a FI port is error disabled, it is effectively shut down and no traffic is sent or received on that port.

The config can be checked/modified as shown in link below

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2/configuring_ports_and_port_channels.html#concept_451A029DEA0D4F2F9E78869D15A8E95E

Cisco Employee

Hi @Michael King,

Hi [@me@mpking.com],

Can you SSH to UCS and send the output of the commands below?

UCS-A# connect nxos b

UCS-B(nxos)# show hardware internal carmel crc

UCS-B(nxos)# show hardware internal carmel eye

UCS-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"

Beginner

Interop-A# connect nxos

I don't think I can get this to display right

Interop-A# connect nxos b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Interop-B(nxos)# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 68 | 68 |
| Eth 1/2 | --- | --- | --- | --- | --- | 58 | 58 |
| Eth 1/3 | --- | --- | --- | --- | --- | 1748 | 1748 |
| Eth 1/4 | --- | --- | --- | --- | --- | 378933 | 378933 |
| Eth 1/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | 252571 | --- | 252571 | --- | --- | --- |
| Eth 1/20 | --- | 129712 | --- | 129710 | --- | --- | --- |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/29 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/30 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/34 | --- | --- | --- | --- | --- | 1648 | 1648 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
Interop-B(nxos)# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 171 mv | 828 mUI | 37/ 35 | 04/28/2016 15:08:37.487235 |a9|ec|e2|2d|90|6c|e2|2d|88|00|
fi0 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:37.492363 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:37.497490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 153 mv | 828 mUI | 31/ 35 | 04/28/2016 15:08:43.594756 |a9|ec|4a|2f|b0|6c|4a|2f|88|00|
fi0 | 200 mv | 828 mUI | 40/ 35 | 04/28/2016 15:08:43.599885 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.605015 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 162 mv | 812 mUI | 34/ 34 | 04/28/2016 15:08:49.707225 |a9|ec|2c|2d|b0|6c|2c|2d|88|00|
fi0 | 200 mv | 921 mUI | 40/ 3b | 04/28/2016 15:08:49.712355 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:49.717481 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 134 mv | 718 mUI | 2b/ 2e | 04/28/2016 15:08:52.787242 |a9|ea|2c|0b|b0|6a|2c|0b|88|00|
fi0 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:52.792370 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:52.797498 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:15.998076 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.028490 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.071352 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.101230 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.131251 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.149960 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.191241 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.209556 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.270749 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.298956 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.324681 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.346728 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 118 mv | 750 mUI | 26/ 30 | 04/28/2016 15:08:37.502645 |a9|dd|9c|25|b0|5d|9c|25|88|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:37.507769 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:37.512900 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 112 mv | 734 mUI | 24/ 2f | 04/28/2016 15:08:43.610162 |a9|da|86|43|92|5a|86|43|88|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:43.615292 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.620417 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 121 mv | 671 mUI | 27/ 2b | 04/28/2016 15:08:49.722629 |a9|d4|a2|67|b0|54|a2|67|88|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:49.727756 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:49.732885 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 121 mv | 796 mUI | 27/ 33 | 04/28/2016 15:08:52.802745 |a9|de|c6|23|f0|5e|c6|23|88|00|
fi0 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:52.807880 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:52.813009 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.505055 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.538961 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.553591 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.574806 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.598739 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.619035 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.644852 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.678867 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.693460 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.719028 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Interop-B(nxos)#
Interop-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
Slot 0 Carmel 2 interrupt statistics:
car_bm3_INT_err_ig_mtu_vio |3 |0 |3 |0
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Interop-B(nxos)#
Cisco Employee

It looks like you're

It looks like you're receiving bad frames from your upstream switches (Eth1/19, Eth1/20) and these are getting stomped and being sent to the IOM (Eth1/1 - Eth1/4).

You need to find out where these are coming from. What kind of switches do you have upstream? If they are Nexus 5548/5596 you can run the same first two commands I sent.

Beginner

It's a Nexus 5548UP (5.1(3)N2

It's a Nexus 5548UP (5.1(3)N2(1c)).  Those ports goto the 2nd Nexus. Ports 11/12

9/10 are a channel group to FI-A (channel group 50)

11/12 are a channel group to FI-B (channel group 51)

There is a VPC-Peer-link on 1/29 and 1/30 (PO1)

and downstream switch on 1/28 (PO2)

IO-Nexus-02# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 403697 | 403697 |
| Eth 1/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/5 | --- | --- | --- | --- | --- | 3056 | 3056 |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | 130434 | --- | --- | 130087 | --- | 1387792 | 1387792 |
| Eth 1/10 | --- | --- | --- | --- | --- | 189038 | 189038 |
| Eth 1/11 | 2537118 | --- | --- | 2532078 | --- | 294708 | 294708 |
| Eth 1/12 | --- | --- | --- | --- | --- | 147652 | 147652 |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | --- | --- | --- | --- | 3144 | 3144 |
| Eth 1/20 | --- | --- | --- | --- | --- | 21917 | 21917 |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | 138 | 138 |
| Eth 1/29 | --- | --- | --- | --- | --- | 3641 | 3641 |
| Eth 1/30 | --- | --- | --- | --- | --- | 96995 | 96995 |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | 586 | 586 |
| Eth 1/34 | --- | --- | --- | --- | --- | 1004 | 1004 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 2/1 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/15 | --- | --- | --- | 1 | --- | --- | --- |
| Eth 2/16 | --- | --- | --- | 1 | --- | --- | --- |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 3/1 | 51746 | --- | --- | 51741 | --- | 16016 | 16016 |
| Eth 3/2 | 51296 | --- | --- | 51282 | --- | 41508 | 41508 |
| Eth 3/3 | 57716 | --- | --- | 57706 | --- | 19237 | 19237 |
| Eth 3/4 | 57536 | --- | --- | 57527 | --- | 10478 | 10478 |
| Eth 3/5 | 61480 | --- | --- | 61471 | --- | 16712 | 16712 |
| Eth 3/6 | 55210 | --- | --- | 55200 | --- | 176914 | 176914 |
| Eth 3/7 | 54444 | --- | --- | 54435 | --- | 163659 | 163659 |
| Eth 3/8 | 60945 | --- | --- | 60935 | --- | 15425 | 15425 |
| Eth 3/9 | 54553 | --- | --- | 54544 | --- | 18428 | 18428 |
| Eth 3/10 | 54639 | --- | --- | 54628 | --- | 24437 | 24437 |
| Eth 3/11 | 2 | --- | --- | 2 | --- | 12940 | 12940 |
| Eth 3/12 | 60311 | --- | --- | 60301 | --- | 8095 | 8095 |
| Eth 3/13 | 56249 | --- | --- | 56239 | --- | 304267 | 304267 |
| Eth 3/14 | 58109 | --- | --- | 58104 | --- | 13854 | 13854 |
| Eth 3/15 | 59819 | --- | --- | 59812 | --- | 27124 | 27124 |
| Eth 3/16 | 49437 | --- | --- | 49432 | --- | 153811 | 153811 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.388864 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.404816 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.420032 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.445690 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 121 mv | 734 mUI | 27/ 2f | 04/29/2016 09:10:00.858343 |a9|e3|a2|01|90|63|a2|01|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.863470 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:10:00.868589 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.475343 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.487316 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.500157 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.471003 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.476130 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 121 mv | 781 mUI | 27/ 32 | 04/29/2016 09:09:45.568212 |a9|da|6e|6a|f0|5a|6e|6a|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:45.573328 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 843 mUI | 40/ 36 | 04/29/2016 09:09:45.578447 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:09:51.673940 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 828 mUI | 40/ 35 | 04/29/2016 09:09:51.679068 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 121 mv | 718 mUI | 27/ 2e | 04/29/2016 09:09:57.768373 |a9|d0|6c|28|f0|50|6c|28|88|00|
fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:57.773503 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.778641 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.560631 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.572713 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.584407 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.596294 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.611372 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.626295 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 59 mv | 562 mUI | 13/ 24 | 04/29/2016 09:09:51.684219 |00|00|00|00|00|00|00|00|00|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:51.689357 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 812 mUI | 40/ 34 | 04/29/2016 09:09:51.694490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 103 mv | 656 mUI | 21/ 2a | 04/29/2016 09:09:57.784242 |a9|de|c0|00|10|5e|c0|00|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:57.789384 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.794516 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.669531 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.684379 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.700335 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.713472 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.726645 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.739353 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.752351 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:42:57.176627 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 131 mv | 750 mUI | 2a/ 30 | 04/29/2016 09:10:00.873831 |a9|e6|ce|19|b2|66|ce|19|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.878952 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:10:00.884073 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 128 mv | 750 mUI | 29/ 30 | 04/29/2016 09:09:21.145872 |a9|df|88|23|d0|5f|88|23|88|00|
fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:21.151004 |00|00|00|00|00|00|00|00|00|00|
fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:21.156141 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.862569 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.903761 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/2 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/4 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/3 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/6 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/5 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/8 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/7 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/9 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/10| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/11| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/12| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/13| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/14| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/15| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/16| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.257487 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.261558 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.265687 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.270909 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.276357 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.280462 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.284353 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.292033 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.298433 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.303185 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.309944 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.316020 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.320194 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.243841 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.247526 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.253393 |00|00|00|00|00|00|00|00|00|00|
fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
car_bm0_INT_err_ig_mtu_vio |1 |0 |1 |0
car_bm2_INT_err_ig_mtu_vio |194 |0 |4 |0
Slot 0 Carmel 2 interrupt statistics:
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Slot 1 Carmel 0 interrupt statistics:
Slot 1 Carmel 1 interrupt statistics:
Slot 2 Carmel 0 interrupt statistics:
Slot 2 Carmel 1 interrupt statistics:
Cisco Employee

Eth1/19 and Eth1/11 are the

Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.

Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|

Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|

N5K bug: CSCuj86736

UCS bug: CSCuo76425

The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.

Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.

Cisco Employee

@Michael King , did the above

[@me@mpking.com] , did the above resolve your issue?

Beginner

Yes, this appears to have

Yes, this appears to have been it.

I see this bug has been patched, so I will schedule a maintenance window to move up.

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards