Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
22032
Views
0
Helpful
9
Replies

TCA: etherNiErrStats crcDelta

Go to solution
Michael King
Level 1
Level 1

‎04-27-2016 07:08 AM - edited ‎03-01-2019 12:42 PM

We recently upgraded our UCS Cluster from 2.2(3f) to 2.2(6e).

I didn't use the auto upgrade, i used set of directions provided to me by TAC.

Since the upgrade, Both of my IOM (2208XP's) throw the following errors:

sys/chassis-1/slot-2/fabric/port-4/fault-F43324

TCA: etherNiErrStats crcDelta, current-value = 54, raised above esc value 5

I've seen it on Slot2/Port4, Slot1/Port1, and Slot1/Port2.

  • Is this a known bug on 2.2(6e)? I can't find anything after searching.
  • Is it new behavior (I didn't see anything in the release notes)
    • Could it be I've always had a problem, and UCS is just now alerting on it?
  • Did the firmware upgrade go sideways, and now I have a ticking time bomb?

Thanks

17 people had this problem
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
Felipe de Mello
Cisco Employee
Cisco Employee
In response to Michael King

‎04-29-2016 07:52 AM

Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.

Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|

Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|

N5K bug: CSCuj86736

UCS bug: CSCuo76425

The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.

Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.

View solution in original post

0 Helpful
9 Replies 9

Go to solution
Wes Austin
Cisco Employee
Cisco Employee

‎04-27-2016 07:26 AM

Hey Michael,

These errors are reporting that there is are CRC errors being logged on your IOM interfaces. The easiest way to try and narrow this down is swapping the cable/optics on those interfaces and monitor to see if they are still being generated.

I believe in the 2.2.6 train we introduced different thresholds for errors in that a fault would be generated once we reach or cross a certain value. If you are unable to isolate the issue, you may opt to open a TAC case to have them investigate further.

Let me know if you need further clarification. 

Wes

0 Helpful

Go to solution
skarthic
Cisco Employee
Cisco Employee

‎04-28-2016 08:24 AM

Michael,

UCSM 2.2.4 adds the following feature to monitor IOM NIF port errors and by default triggers a warning if the number of errors increase the value of 5. You can use this feature to automatically shut down the port if the CRCs go beyond a particular value.

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html#pgfId-282711

Policy-Based Port Error Handling—If Cisco UCS Manager detects any errors on active IOM Network Interface (NIF) ports, and if the Error-disable setting is enabled, Cisco UCS Manager automatically disables the respective FI port that is connected to the IOM NIF port that had errors.

When a FI port is error disabled, it is effectively shut down and no traffic is sent or received on that port.

The config can be checked/modified as shown in link below

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2/configuring_ports_and_port_channels.html#concept_451A029DEA0D4F2F9E78869D15A8E95E

0 Helpful

Go to solution
Felipe de Mello
Cisco Employee
Cisco Employee

‎04-28-2016 10:23 AM

Hi [@me@mpking.com],

Can you SSH to UCS and send the output of the commands below?

UCS-A# connect nxos b

UCS-B(nxos)# show hardware internal carmel crc

UCS-B(nxos)# show hardware internal carmel eye

UCS-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"

0 Helpful

Go to solution
Michael King
Level 1
Level 1
In response to Felipe de Mello

‎04-28-2016 12:16 PM

I don't think I can get this to display right

Interop-A# connect nxos b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2015, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Interop-B(nxos)# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 68 | 68 |
| Eth 1/2 | --- | --- | --- | --- | --- | 58 | 58 |
| Eth 1/3 | --- | --- | --- | --- | --- | 1748 | 1748 |
| Eth 1/4 | --- | --- | --- | --- | --- | 378933 | 378933 |
| Eth 1/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | 252571 | --- | 252571 | --- | --- | --- |
| Eth 1/20 | --- | 129712 | --- | 129710 | --- | --- | --- |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/29 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/30 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/34 | --- | --- | --- | --- | --- | 1648 | 1648 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
Interop-B(nxos)# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 171 mv | 828 mUI | 37/ 35 | 04/28/2016 15:08:37.487235 |a9|ec|e2|2d|90|6c|e2|2d|88|00|
 fi0 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:37.492363 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:37.497490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 153 mv | 828 mUI | 31/ 35 | 04/28/2016 15:08:43.594756 |a9|ec|4a|2f|b0|6c|4a|2f|88|00|
 fi0 | 200 mv | 828 mUI | 40/ 35 | 04/28/2016 15:08:43.599885 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.605015 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 162 mv | 812 mUI | 34/ 34 | 04/28/2016 15:08:49.707225 |a9|ec|2c|2d|b0|6c|2c|2d|88|00|
 fi0 | 200 mv | 921 mUI | 40/ 3b | 04/28/2016 15:08:49.712355 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 890 mUI | 40/ 39 | 04/28/2016 15:08:49.717481 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 134 mv | 718 mUI | 2b/ 2e | 04/28/2016 15:08:52.787242 |a9|ea|2c|0b|b0|6a|2c|0b|88|00|
 fi0 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:52.792370 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:52.797498 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:15.998076 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.028490 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.071352 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.101230 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.131251 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.149960 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.191241 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.209556 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.270749 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.298956 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.324681 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.346728 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 118 mv | 750 mUI | 26/ 30 | 04/28/2016 15:08:37.502645 |a9|dd|9c|25|b0|5d|9c|25|88|00|
 fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:37.507769 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:37.512900 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 112 mv | 734 mUI | 24/ 2f | 04/28/2016 15:08:43.610162 |a9|da|86|43|92|5a|86|43|88|00|
 fi0 | 200 mv | 906 mUI | 40/ 3a | 04/28/2016 15:08:43.615292 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 843 mUI | 40/ 36 | 04/28/2016 15:08:43.620417 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 121 mv | 671 mUI | 27/ 2b | 04/28/2016 15:08:49.722629 |a9|d4|a2|67|b0|54|a2|67|88|00|
 fi0 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:49.727756 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 875 mUI | 40/ 38 | 04/28/2016 15:08:49.732885 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 121 mv | 796 mUI | 27/ 33 | 04/28/2016 15:08:52.802745 |a9|de|c6|23|f0|5e|c6|23|88|00|
 fi0 | 200 mv | 812 mUI | 40/ 34 | 04/28/2016 15:08:52.807880 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 859 mUI | 40/ 37 | 04/28/2016 15:08:52.813009 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.505055 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.538961 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.553591 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.574806 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.598739 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.619035 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.644852 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.678867 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.693460 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:16.719028 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Interop-B(nxos)#
Interop-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
Slot 0 Carmel 2 interrupt statistics:
car_bm3_INT_err_ig_mtu_vio |3 |0 |3 |0
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Interop-B(nxos)#
0 Helpful

Go to solution
Felipe de Mello
Cisco Employee
Cisco Employee
In response to Michael King

‎04-28-2016 05:48 PM

It looks like you're receiving bad frames from your upstream switches (Eth1/19, Eth1/20) and these are getting stomped and being sent to the IOM (Eth1/1 - Eth1/4).

You need to find out where these are coming from. What kind of switches do you have upstream? If they are Nexus 5548/5596 you can run the same first two commands I sent.

0 Helpful

Go to solution
Michael King
Level 1
Level 1
In response to Felipe de Mello

‎04-29-2016 06:15 AM

It's a Nexus 5548UP (5.1(3)N2(1c)).  Those ports goto the 2nd Nexus. Ports 11/12

9/10 are a channel group to FI-A (channel group 50)

11/12 are a channel group to FI-B (channel group 51)

There is a VPC-Peer-link on 1/29 and 1/30 (PO1)

and downstream switch on 1/28 (PO2)

IO-Nexus-02# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | 403697 | 403697 |
| Eth 1/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/5 | --- | --- | --- | --- | --- | 3056 | 3056 |
| Eth 1/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | 130434 | --- | --- | 130087 | --- | 1387792 | 1387792 |
| Eth 1/10 | --- | --- | --- | --- | --- | 189038 | 189038 |
| Eth 1/11 | 2537118 | --- | --- | 2532078 | --- | 294708 | 294708 |
| Eth 1/12 | --- | --- | --- | --- | --- | 147652 | 147652 |
| Eth 1/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/15 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/16 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/17 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/18 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/19 | --- | --- | --- | --- | --- | 3144 | 3144 |
| Eth 1/20 | --- | --- | --- | --- | --- | 21917 | 21917 |
| Eth 1/21 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/22 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/23 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/24 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/25 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/26 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/27 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/28 | --- | --- | --- | --- | --- | 138 | 138 |
| Eth 1/29 | --- | --- | --- | --- | --- | 3641 | 3641 |
| Eth 1/30 | --- | --- | --- | --- | --- | 96995 | 96995 |
| Eth 1/31 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/32 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/33 | --- | --- | --- | --- | --- | 586 | 586 |
| Eth 1/34 | --- | --- | --- | --- | --- | 1004 | 1004 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 2/1 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/6 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/9 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/10 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/11 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/12 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/13 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/14 | --- | --- | --- | --- | --- | --- | --- |
| Eth 2/15 | --- | --- | --- | 1 | --- | --- | --- |
| Eth 2/16 | --- | --- | --- | 1 | --- | --- | --- |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 3/1 | 51746 | --- | --- | 51741 | --- | 16016 | 16016 |
| Eth 3/2 | 51296 | --- | --- | 51282 | --- | 41508 | 41508 |
| Eth 3/3 | 57716 | --- | --- | 57706 | --- | 19237 | 19237 |
| Eth 3/4 | 57536 | --- | --- | 57527 | --- | 10478 | 10478 |
| Eth 3/5 | 61480 | --- | --- | 61471 | --- | 16712 | 16712 |
| Eth 3/6 | 55210 | --- | --- | 55200 | --- | 176914 | 176914 |
| Eth 3/7 | 54444 | --- | --- | 54435 | --- | 163659 | 163659 |
| Eth 3/8 | 60945 | --- | --- | 60935 | --- | 15425 | 15425 |
| Eth 3/9 | 54553 | --- | --- | 54544 | --- | 18428 | 18428 |
| Eth 3/10 | 54639 | --- | --- | 54628 | --- | 24437 | 24437 |
| Eth 3/11 | 2 | --- | --- | 2 | --- | 12940 | 12940 |
| Eth 3/12 | 60311 | --- | --- | 60301 | --- | 8095 | 8095 |
| Eth 3/13 | 56249 | --- | --- | 56239 | --- | 304267 | 304267 |
| Eth 3/14 | 58109 | --- | --- | 58104 | --- | 13854 | 13854 |
| Eth 3/15 | 59819 | --- | --- | 59812 | --- | 27124 | 27124 |
| Eth 3/16 | 49437 | --- | --- | 49432 | --- | 153811 | 153811 |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
+----------+------------+------------+------------+------------+------------+------------+------------+
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel eye
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
| Port | Eye Height | Eye Width | Raw values | Time measured |St|20|21|22|23|24|25|26|2E|2F|
+-------+------------+-------------+------------+----------------------------+--+--+--+--+--+--+--+--+--+--+
Eth 1/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.388864 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.404816 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.420032 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.445690 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/5 | 121 mv | 734 mUI | 27/ 2f | 04/29/2016 09:10:00.858343 |a9|e3|a2|01|90|63|a2|01|88|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.863470 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:10:00.868589 |00|00|00|00|00|00|00|00|00|00|
Eth 1/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.475343 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.487316 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.500157 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.471003 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:39.476130 |00|00|00|00|00|00|00|00|00|00|
Eth 1/10| 121 mv | 781 mUI | 27/ 32 | 04/29/2016 09:09:45.568212 |a9|da|6e|6a|f0|5a|6e|6a|88|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:45.573328 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 843 mUI | 40/ 36 | 04/29/2016 09:09:45.578447 |00|00|00|00|00|00|00|00|00|00|
Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:09:51.673940 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 828 mUI | 40/ 35 | 04/29/2016 09:09:51.679068 |00|00|00|00|00|00|00|00|00|00|
Eth 1/12| 121 mv | 718 mUI | 27/ 2e | 04/29/2016 09:09:57.768373 |a9|d0|6c|28|f0|50|6c|28|88|00|
 fi0 | 200 mv | 859 mUI | 40/ 37 | 04/29/2016 09:09:57.773503 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.778641 |00|00|00|00|00|00|00|00|00|00|
Eth 1/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.560631 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.572713 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.584407 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.596294 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/17| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.611372 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/18| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.626295 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/19| 59 mv | 562 mUI | 13/ 24 | 04/29/2016 09:09:51.684219 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:51.689357 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 812 mUI | 40/ 34 | 04/29/2016 09:09:51.694490 |00|00|00|00|00|00|00|00|00|00|
Eth 1/20| 103 mv | 656 mUI | 21/ 2a | 04/29/2016 09:09:57.784242 |a9|de|c0|00|10|5e|c0|00|88|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:57.789384 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:57.794516 |00|00|00|00|00|00|00|00|00|00|
Eth 1/21| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.669531 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/22| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.684379 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/23| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.700335 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/24| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.713472 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/25| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.726645 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/26| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.739353 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/27| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.752351 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/28| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:42:57.176627 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/29| 131 mv | 750 mUI | 2a/ 30 | 04/29/2016 09:10:00.873831 |a9|e6|ce|19|b2|66|ce|19|88|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:10:00.878952 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 906 mUI | 40/ 3a | 04/29/2016 09:10:00.884073 |00|00|00|00|00|00|00|00|00|00|
Eth 1/30| 128 mv | 750 mUI | 29/ 30 | 04/29/2016 09:09:21.145872 |a9|df|88|23|d0|5f|88|23|88|00|
 fi0 | 200 mv | 890 mUI | 40/ 39 | 04/29/2016 09:09:21.151004 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 200 mv | 875 mUI | 40/ 38 | 04/29/2016 09:09:21.156141 |00|00|00|00|00|00|00|00|00|00|
Eth 1/31| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.862569 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/32| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:39:31.903761 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/33| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 1/34| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/2 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/4 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/3 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/6 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/5 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/8 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/7 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/9 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/10| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/11| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/12| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/13| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/14| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/15| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 2/16| 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/1 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.257487 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/2 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.261558 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/3 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.265687 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/4 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.270909 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/5 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.276357 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/6 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.280462 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/7 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.284353 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/8 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.292033 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/9 | 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.298433 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/10| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.303185 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/11| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.309944 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/12| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.316020 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/13| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.320194 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/14| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.243841 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/15| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.247526 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
Eth 3/16| 0 mv | 0 mUI | 0/ 0 | 04/21/2016 17:40:48.253393 |00|00|00|00|00|00|00|00|00|00|
 fi0 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
 fi1 | 0 mv | 0 mUI | 0/ 0 | 12/31/1969 19:00:00.000000 |00|00|00|00|00|00|00|00|00|00|
IO-Nexus-02#
IO-Nexus-02# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"
Slot 0 Carmel 0 interrupt statistics:
Slot 0 Carmel 1 interrupt statistics:
car_bm0_INT_err_ig_mtu_vio |1 |0 |1 |0
car_bm2_INT_err_ig_mtu_vio |194 |0 |4 |0
Slot 0 Carmel 2 interrupt statistics:
Slot 0 Carmel 3 interrupt statistics:
Slot 0 Carmel 4 interrupt statistics:
Slot 1 Carmel 0 interrupt statistics:
Slot 1 Carmel 1 interrupt statistics:
Slot 2 Carmel 0 interrupt statistics:
Slot 2 Carmel 1 interrupt statistics:
0 Helpful

Go to solution
Felipe de Mello
Cisco Employee
Cisco Employee
In response to Michael King

‎04-29-2016 07:52 AM

Eth1/19 and Eth1/11 are the problem interfaces causing the CRCs. As you can see, their voltage is low.

Eth 1/9 | 37 mv | 390 mUI | c/ 19 | 04/29/2016 09:09:39.465875 |00|00|00|00|00|00|00|00|00|00|

Eth 1/11| 37 mv | 359 mUI | c/ 17 | 04/29/2016 09:09:51.668804 |00|00|00|00|00|00|00|00|00|00|

N5K bug: CSCuj86736

UCS bug: CSCuo76425

The workaround is to flap the affected interfaces and check show hardware internal carmel eye to make sure the voltage increases and then you can use show hardware internal carmel crc to make sure the CRCs have stopped.

Though the interfaces are in a port-channel, if you can't take any traffic disruptions the flapping should be done in a maintenance window.

0 Helpful

Go to solution
Felipe de Mello
Cisco Employee
Cisco Employee
In response to Felipe de Mello

‎05-02-2016 06:09 PM

[@me@mpking.com] , did the above resolve your issue?

0 Helpful

Go to solution
Michael King
Level 1
Level 1
In response to Felipe de Mello

‎05-04-2016 06:27 AM

Yes, this appears to have been it.

I see this bug has been patched, so I will schedule a maintenance window to move up.

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents

Review Cisco Networking products for a $25 gift card