Re: GLC-SX-MMD / WS-C2960X-24PS-L/ input errors and CRC

Peter Nesta · ‎02-22-2019

Hi everyone,

I hope you can help me. Currently I have strange problems with one of our cisco 2960x stack.

The 2960x consisting of 3 members has a total of 8 fibre optic connections to 4 x 2960x access switches. So each access switch is connected to the main stack in a port channel. Now we had several times the problem that suddenly all fiber optic connections failed. UDLD error were detected and CRC errors and input errors are visible on all uplinks.

I have attached the logs and the configuration of the interfaces:

001653: Feb 22 16:01:44.387 GMT: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.0055.0c82 in vlan 38 is flapping between port Po2 and port Gi1/0/18
001654: Feb 22 16:01:53.276 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/25 left the port-channel Po2
001655: Feb 22 16:01:53.590 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3
001656: Feb 22 16:01:54.262 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/25, changed state to down
001657: 000056: Feb 22 16:01:53.271 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/25 left the port-channel Po2 (XXX-SWT01-2)
001658: 000057: Feb 22 16:01:53.586 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3 (XXX-SWT01-2)
001659: 000051: Feb 22 16:01:53.271 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/25 left the port-channel Po2 (XXX-SWT01-3)
001660: 000052: Feb 22 16:01:53.589 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3 (XXX-SWT01-3)
001661: Feb 22 16:01:54.580 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/26, changed state to down
001662: Feb 22 16:01:57.005 GMT: %EC-5-BUNDLE: Interface Gi2/0/25 joined port-channel Po2
001663: Feb 22 16:01:57.226 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3
001664: 000058: Feb 22 16:01:57.004 GMT: %EC-5-BUNDLE: Interface Gi2/0/25 joined port-channel Po2 (XXX-SWT01-2)
001665: 000059: Feb 22 16:01:57.221 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3 (XXX-SWT01-2)
001666: 000053: Feb 22 16:01:57.004 GMT: %EC-5-BUNDLE: Interface Gi2/0/25 joined port-channel Po2 (XXX-SWT01-3)
001667: 000054: Feb 22 16:01:57.221 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3 (XXX-SWT01-3)
001668: Feb 22 16:01:59.652 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4
001669: Feb 22 16:02:00.637 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/27, changed state to down
001670: 000060: Feb 22 16:01:59.647 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4 (XXX-SWT01-2)
001671: 000055: Feb 22 16:01:59.657 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4 (XXX-SWT01-3)
001672: Feb 22 16:02:01.801 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/26, changed state to up
001673: Feb 22 16:02:01.871 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/25, changed state to up
001674: Feb 22 16:02:03.231 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4
001675: 000061: Feb 22 16:02:03.229 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4 (XXX-SWT01-2)
001676: 000056: Feb 22 16:02:03.229 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4 (XXX-SWT01-3)
001677: Feb 22 16:02:08.208 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/27, changed state to up
001678: Feb 22 16:02:09.760 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to down
001679: Feb 22 16:02:11.588 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5
001680: 000062: Feb 22 16:02:11.587 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5 (XXX-SWT01-2)
001681: 000057: Feb 22 16:02:11.587 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5 (XXX-SWT01-3)
001682: Feb 22 16:02:19.076 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5
001683: 000058: Feb 22 16:02:19.073 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5 (XXX-SWT01-3)
001684: Feb 22 16:02:20.086 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to up
001685: 000063: Feb 22 16:02:19.073 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5 (XXX-SWT01-2)
001686: Feb 22 16:03:05.201 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/27, changed state to down
001687: Feb 22 16:03:06.467 GMT: %UDLD-4-UDLD_PORT_DISABLED: UDLD disabled interface Gi1/0/27, neighbor mismatch detected
001688: Feb 22 16:03:06.467 GMT: %PM-4-ERR_DISABLE: udld error detected on Gi1/0/27, putting Gi1/0/27 in err-disable state
001689: Feb 22 16:03:06.467 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/27 left the port-channel Po4
001690: Feb 22 16:03:06.970 GMT: %DTP-5-NONTRUNKPORTON: Port Gi1/0/27 has become non-trunk
001691: 000064: Feb 22 16:03:06.469 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/27 left the port-channel Po4 (XXX-SWT01-2)
001692: 000059: Feb 22 16:03:06.469 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/27 left the port-channel Po4 (XXX-SWT01-3)
001693: Feb 22 16:03:08.473 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/27, changed state to down
001694: Feb 22 16:03:09.396 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/26 left the port-channel Po3
001695: Feb 22 16:03:10.382 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/26, changed state to down
001696: 000065: Feb 22 16:03:09.394 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/26 left the port-channel Po3 (XXX-SWT01-2)
001697: 000060: Feb 22 16:03:09.394 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/26 left the port-channel Po3 (XXX-SWT01-3)
001698: Feb 22 16:03:10.930 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/25 left the port-channel Po2
001699: Feb 22 16:03:11.920 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/25, changed state to down
001700: 000061: Feb 22 16:03:10.929 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/25 left the port-channel Po2 (XXX-SWT01-3)
001701: 000066: Feb 22 16:03:10.939 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/25 left the port-channel Po2 (XXX-SWT01-2)
001702: Feb 22 16:03:13.202 GMT: %EC-5-BUNDLE: Interface Gi1/0/26 joined port-channel Po3
001703: 000067: Feb 22 16:03:13.197 GMT: %EC-5-BUNDLE: Interface Gi1/0/26 joined port-channel Po3 (XXX-SWT01-2)
001704: 000062: Feb 22 16:03:13.197 GMT: %EC-5-BUNDLE: Interface Gi1/0/26 joined port-channel Po3 (XXX-SWT01-3)
001705: Feb 22 16:03:14.524 GMT: %EC-5-BUNDLE: Interface Gi1/0/25 joined port-channel Po2
001706: 000063: Feb 22 16:03:14.518 GMT: %EC-5-BUNDLE: Interface Gi1/0/25 joined port-channel Po2 (XXX-SWT01-3)
001707: 000068: Feb 22 16:03:14.525 GMT: %EC-5-BUNDLE: Interface Gi1/0/25 joined port-channel Po2 (XXX-SWT01-2)
001708: Feb 22 16:03:17.638 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/26, changed state to up
001709: Feb 22 16:03:19.288 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/25, changed state to up
001710: Feb 22 16:03:50.845 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/28 left the port-channel Po5
001711: Feb 22 16:03:51.390 GMT: %DTP-5-NONTRUNKPORTON: Port Gi1/0/28 has become non-trunk
001712: Feb 22 16:03:51.841 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/28, changed state to down
001713: 000069: Feb 22 16:03:50.844 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/28 left the port-channel Po5 (XXX-SWT01-2)
001714: 000064: Feb 22 16:03:50.851 GMT: %EC-5-UNBUNDLE: Interface Gi1/0/28 left the port-channel Po5 (XXX-SWT01-3)
001715: Feb 22 16:03:52.890 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/28, changed state to down
001716: Feb 22 16:04:14.582 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3
001717: 000070: Feb 22 16:04:13.615 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3 (XXX-SWT01-2)
001718: Feb 22 16:04:14.628 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/26, changed state to down
001719: 000065: Feb 22 16:04:14.580 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/26 left the port-channel Po3 (XXX-SWT01-3)
001720: Feb 22 16:04:17.417 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3
001729: Feb 22 16:04:17.896 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4
001730: 000071: Feb 22 16:04:17.404 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3 (XXX-SWT01-2)
001731: 000072: Feb 22 16:04:17.869 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4 (XXX-SWT01-2)
001732: 000066: Feb 22 16:04:17.415 GMT: %EC-5-BUNDLE: Interface Gi2/0/26 joined port-channel Po3 (XXX-SWT01-3)
001733: 000067: Feb 22 16:04:17.894 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/27 left the port-channel Po4 (XXX-SWT01-3)
001734: Feb 22 16:04:18.850 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/27, changed state to down
001735: Feb 22 16:04:18.854 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to down
001736: Feb 22 16:04:19.857 GMT: %LINK-3-UPDOWN: Interface Port-channel4, changed state to down
001745: 000073: Feb 22 16:04:21.417 GMT: %UDLD-4-UDLD_PORT_DISABLED: UDLD disabled interface Gi2/0/28, unidirectional link detected (XXX-SWT01-2)
001746: 000074: Feb 22 16:04:21.417 GMT: %PM-4-ERR_DISABLE: udld error detected on Gi2/0/28, putting Gi2/0/28 in err-disable state (XXX-SWT01-2)
001747: Feb 22 16:04:21.465 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5
001748: Feb 22 16:04:21.625 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4
001749: Feb 22 16:04:21.681 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/26, changed state to up
001750: Feb 22 16:04:22.433 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to down
001751: Feb 22 16:04:22.433 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel5, changed state to down
001752: 000075: Feb 22 16:04:21.424 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5 (XXX-SWT01-2)
001753: 000076: Feb 22 16:04:21.616 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4 (XXX-SWT01-2)
001754: 000077: Feb 22 16:04:21.927 GMT: %DTP-5-NONTRUNKPORTON: Port Gi2/0/28 has become non-trunk (XXX-SWT01-2)
001755: 000068: Feb 22 16:04:21.476 GMT: %EC-5-UNBUNDLE: Interface Gi2/0/28 left the port-channel Po5 (XXX-SWT01-3)
001756: 000069: Feb 22 16:04:21.630 GMT: %EC-5-BUNDLE: Interface Gi2/0/27 joined port-channel Po4 (XXX-SWT01-3)
001757: Feb 22 16:04:23.436 GMT: %LINK-3-UPDOWN: Interface Port-channel5, changed state to down
001758: Feb 22 16:04:23.464 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/28, changed state to down
001775: Feb 22 16:04:26.617 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/27, changed state to up
001776: Feb 22 16:04:27.599 GMT: %LINK-3-UPDOWN: Interface Port-channel4, changed state to up
001777: Feb 22 16:04:28.602 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to up
001778: Feb 22 16:05:06.472 GMT: %PM-4-ERR_RECOVER: Attempting to recover from udld err-disable state on Gi1/0/27
001779: Feb 22 16:05:08.520 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/27, changed state to up
001780: Feb 22 16:05:10.020 GMT: %DTP-5-TRUNKPORTON: Port Gi1/0/27 has become dot1q trunk
001781: Feb 22 16:05:13.250 GMT: %EC-5-BUNDLE: Interface Gi1/0/27 joined port-channel Po4
001782: 000070: Feb 22 16:05:13.244 GMT: %EC-5-BUNDLE: Interface Gi1/0/27 joined port-channel Po4 (XXX-SWT01-3)
001783: 000078: Feb 22 16:05:13.244 GMT: %EC-5-BUNDLE: Interface Gi1/0/27 joined port-channel Po4 (XXX-SWT01-2)
001784: Feb 22 16:05:14.340 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/27, changed state to up
001785: Feb 22 16:05:52.842 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/28, changed state to up
001786: Feb 22 16:05:54.342 GMT: %DTP-5-TRUNKPORTON: Port Gi1/0/28 has become dot1q trunk
001787: Feb 22 16:05:57.348 GMT: %EC-5-BUNDLE: Interface Gi1/0/28 joined port-channel Po5
001788: 000079: Feb 22 16:05:57.343 GMT: %EC-5-BUNDLE: Interface Gi1/0/28 joined port-channel Po5 (XXX-SWT01-2)
001789: 000071: Feb 22 16:05:57.347 GMT: %EC-5-BUNDLE: Interface Gi1/0/28 joined port-channel Po5 (XXX-SWT01-3)
001806: Feb 22 16:05:59.484 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/28, changed state to up
001807: Feb 22 16:06:00.466 GMT: %LINK-3-UPDOWN: Interface Port-channel5, changed state to up
001808: Feb 22 16:06:01.466 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel5, changed state to up
001809: 000080: Feb 22 16:06:21.433 GMT: %PM-4-ERR_RECOVER: Attempting to recover from udld err-disable state on Gi2/0/28 (XXX-SWT01-2)
001810: Feb 22 16:06:23.490 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/28, changed state to up
001811: 000081: Feb 22 16:06:24.991 GMT: %DTP-5-TRUNKPORTON: Port Gi2/0/28 has become dot1q trunk (XXX-SWT01-2)
001812: Feb 22 16:06:28.171 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5
001813: 000072: Feb 22 16:06:28.168 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5 (XXX-SWT01-3)
001814: 000082: Feb 22 16:06:28.164 GMT: %EC-5-BUNDLE: Interface Gi2/0/28 joined port-channel Po5 (XXX-SWT01-2)
001815: Feb 22 16:06:29.377 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to up
001816: Feb 22 16:09:48.354 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to down
001817: Feb 22 16:09:49.361 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to down
001818: Feb 22 16:09:53.153 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to up
001819: Feb 22 16:09:54.153 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to up
001820: Feb 22 16:34:54.141 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to down
001821: Feb 22 16:34:55.147 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to down
001822: Feb 22 16:34:57.895 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to up
001823: Feb 22 16:34:58.898 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to up
001824: Feb 22 16:52:35.410 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to down
001825: Feb 22 16:52:36.413 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to down
001826: Feb 22 16:52:44.086 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to up
001827: Feb 22 16:52:45.092 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to up
001828: Feb 22 16:59:52.007 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to down
001829: Feb 22 16:59:53.014 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to down
001830: Feb 22 17:00:00.728 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to up
001831: Feb 22 17:00:01.731 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to up
001832: Feb 22 17:07:36.774 GMT: %SSH-5-SSH2_SESSION: SSH2 Session request from 172.19.16.29 (tty = 0) using crypto cipher 'aes256-ctr', hmac 'hmac-sha1' Succeeded
001833: Feb 22 17:07:42.342 GMT: %SSH-5-SSH2_USERAUTH: User '' authentication for SSH2 Session from 172.19.16.29 (tty = 0) using crypto cipher 'aes256-ctr', hmac 'hmac-sha1' Succeeded
001834: Feb 22 17:19:46.910 GMT: %SSH-5-SSH2_CLOSE: SSH2 Session from 172.19.16.29 (tty = 0) for user '' using crypto cipher 'aes256-ctr', hmac 'hmac-sha1' closed
001835: Feb 22 17:21:33.804 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to down
001836: Feb 22 17:21:34.807 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to down
001837: Feb 22 17:21:38.603 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to up
001838: Feb 22 17:21:39.603 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to up
001839: Feb 22 17:23:33.613 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to down
001840: Feb 22 17:23:34.613 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to down
001841: Feb 22 17:23:37.329 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/12, changed state to up
001842: Feb 22 17:23:38.332 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to up
001843: Feb 22 18:10:29.815 GMT: %SSH-5-SSH_CLOSE: SSH Session from 172.20.2.136 (tty = 0) for user '' using crypto cipher '' closed
001844: Feb 22 18:10:55.003 GMT: %SSH-5-SSH2_SESSION: SSH2 Session request from 172.20.5.94 (tty = 0) using crypto cipher 'aes128-ctr', hmac 'hmac-sha1' Succeeded
001845: Feb 22 18:10:57.594 GMT: %SSH-5-SSH2_USERAUTH: User '' authentication for SSH2 Session from 172.20.5.94 (tty = 0) using crypto cipher 'aes128-ctr', hmac 'hmac-sha1' Succeeded
001846: Feb 22 18:37:04.781 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/9, changed state to down
001847: Feb 22 18:37:05.780 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/9, changed state to down
001848: Feb 22 18:37:13.453 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/9, changed state to up
001849: Feb 22 18:37:14.456 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/9, changed state to up
001850: Feb 22 18:44:05.627 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/15, changed state to down
001851: Feb 22 18:44:06.627 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/15, changed state to down
001852: Feb 22 18:44:16.022 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/15, changed state to up
001853: Feb 22 18:44:17.026 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/15, changed state to up
001854: Feb 22 18:48:38.271 GMT: %SSH-5-SSH2_CLOSE: SSH2 Session from 172.20.5.94 (tty = 0) for user '' using crypto cipher 'aes128-ctr', hmac 'hmac-sha1' closed
001855: Feb 22 19:04:39.707 GMT: %SSH-5-SSH2_SESSION: SSH2 Session request from 172.20.5.94 (tty = 0) using crypto cipher 'aes128-ctr', hmac 'hmac-sha1' Succeeded
001856: Feb 22 19:04:48.127 GMT: %SSH-5-SSH2_USERAUTH: User '' authentication for SSH2 Session from 172.20.5.94 (tty = 0) using crypto cipher 'aes128-ctr', hmac 'hmac-sha1' Succeeded
001857: Feb 22 19:07:07.560 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/9, changed state to down
001858: Feb 22 19:07:08.563 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/9, changed state to down
001859: Feb 22 19:07:11.310 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/9, changed state to up
001860: Feb 22 19:07:12.317 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/9, changed state to up

sh int gigabitEthernet 1/0/25
GigabitEthernet1/0/25 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 9c57.addf.4e99 (bia 9c57.addf.4e99)
  Description: DE-EXK-01-SWT02 Gig1/0/25
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:04, output 00:00:00, output hang never
  Last clearing of "show interface" counters 3d11h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 166103787
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 67000 bits/sec, 59 packets/sec
  5 minute output rate 1436000 bits/sec, 1233 packets/sec
     15501143 packets input, 10659469325 bytes, 0 no buffer
     Received 66630 broadcasts (57430 multicasts)
     0 runts, 0 giants, 0 throttles 
     66262274 input errors, 66262274 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 57430 multicast, 0 pause input
     0 input packets with dribble condition detected
     124507244 packets output, 16970911615 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

sh int gigabitEthernet 2/0/25
GigabitEthernet2/0/25 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is 5006.ab01.8d19 (bia 5006.ab01.8d19)
  Description: DE-EXK-01-SWT02 Gig1/0/27
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:02, output 00:00:03, output hang never
  Last clearing of "show interface" counters 3d11h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 272832816
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 299000 bits/sec, 252 packets/sec
  5 minute output rate 452000 bits/sec, 424 packets/sec
     71083144 packets input, 23595688597 bytes, 0 no buffer
     Received 1509755 broadcasts (77293 multicasts)
     0 runts, 0 giants, 0 throttles 
     104904871 input errors, 104904871 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 77293 multicast, 0 pause input
     0 input packets with dribble condition detected
     492605297 packets output, 68476425622 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Configuration:

All switches are configured in the same way.

h run int gigabitEthernet 1/0/25
Building configuration...

Current configuration : 552 bytes
!
interface GigabitEthernet1/0/25
 description SWT02 Gig1/0/25
 switchport trunk native vlan 100
 switchport trunk allowed vlan 10,11,20,38,39,62,100,900
 switchport mode trunk
 logging event trunk-status
 logging event bundle-status
 logging event status
 srr-queue bandwidth share 10 10 50 30
 queue-set 2
 priority-queue out 
 udld port
 mls qos trust dscp
 spanning-tree portfast disable
 spanning-tree bpdufilter disable
 spanning-tree bpduguard disable
 spanning-tree guard none
 channel-protocol lacp
 channel-group 2 mode active

##########################

nterface GigabitEthernet2/0/25
 description SWT02 Gig1/0/27
 switchport trunk native vlan 100
 switchport trunk allowed vlan 10,11,20,38,39,62,100,900
 switchport mode trunk
 logging event trunk-status
 logging event bundle-status
 logging event status
 srr-queue bandwidth share 10 10 50 30
 queue-set 2
 priority-queue out 
 udld port
 mls qos trust dscp
 spanning-tree portfast disable
 spanning-tree bpdufilter disable
 spanning-tree bpduguard disable
 spanning-tree guard none
 channel-protocol lacp
 channel-group 2 mode active
end

interface Port-channel2
 description 2G CHANNEL SWT02 Gig1/0/25, Gig1/0/27
 switchport trunk native vlan 100
 switchport trunk allowed vlan 10,11,20,38,39,62,100,900
 switchport mode trunk
 logging event trunk-status
 logging event bundle-status
 logging event status
 spanning-tree portfast disable
 spanning-tree bpdufilter disable
 spanning-tree bpduguard disable
 spanning-tree guard none
end

What do you think that might be? Not all SFPs can be defective. They are also different fiber optic lines to each access switch.

IOS Version is:

Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(2)E7,

balaji.bandi · ‎02-22-2019

There is a reason for UDLD to detect a link failure, and you should know what it is so that it can be fixed. Make sure the connectors are securely fastened and double check configs on the fiber port.

Make sure you disable udld recovery it is not an good practice, i can see your log it keep recovery from UDLD disable port back to enable, this will have impact on network.

1. make sure you check all the cables

2. SFP.

3. can show post show vlan from both the switches ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Peter Nesta · ‎02-22-2019

Thanks for your post.

I have also replaced the SFPs but without success.

SWT01#sh vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi1/0/1, Gi2/0/1, Gi3/0/25, Gi3/0/26, Gi3/0/27, Gi3/0/28
2    xxxx                          active    
3    xxxx                  active    Gi1/0/6, Gi1/0/8, Gi2/0/6, Gi2/0/8
4   xxxx                   active    Gi1/0/3, Gi1/0/4, Gi2/0/3, Gi2/0/4
10   xxxx               active    Gi1/0/10, Gi1/0/11, Gi2/0/9, Gi2/0/10, Gi2/0/11, Gi2/0/12, Gi2/0/17, Gi2/0/19, Gi2/0/22, Gi2/0/23, Gi2/0/24, Gi3/0/1, Gi3/0/9, Gi3/0/13, Gi3/0/16
                                                Gi3/0/19, Gi3/0/22
11   xxxx            active    Gi1/0/12, Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Gi1/0/17, Gi2/0/13, Gi2/0/14, Gi2/0/15, Gi2/0/16, Gi2/0/18, Gi2/0/20, Gi2/0/21, Gi3/0/2, Gi3/0/3
                                                Gi3/0/4, Gi3/0/5, Gi3/0/6, Gi3/0/7, Gi3/0/8, Gi3/0/10, Gi3/0/11, Gi3/0/12, Gi3/0/14, Gi3/0/15, Gi3/0/17, Gi3/0/20, Gi3/0/21, Gi3/0/23, Gi3/0/24
20   xxxxx             active    Gi1/0/5, Gi1/0/12, Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Gi1/0/17, Gi2/0/5, Gi2/0/13, Gi2/0/14, Gi2/0/15, Gi2/0/16, Gi2/0/17, Gi2/0/18, Gi2/0/19
                                                Gi2/0/20, Gi2/0/21, Gi2/0/22, Gi2/0/24, Gi3/0/1, Gi3/0/2, Gi3/0/3, Gi3/0/4, Gi3/0/5, Gi3/0/6, Gi3/0/7, Gi3/0/8, Gi3/0/9, Gi3/0/10, Gi3/0/11, Gi3/0/12
                                                Gi3/0/13, Gi3/0/14, Gi3/0/15, Gi3/0/16, Gi3/0/17, Gi3/0/19, Gi3/0/20, Gi3/0/21, Gi3/0/22, Gi3/0/23, Gi3/0/24
38   xxxx             active    
39   xxx             active    
60   xxxxx              active    
62   xxxxxxxx                        active    
63   xxxxxxx                      active    
100  xxxxxxx             active    Gi1/0/7, Gi2/0/7
900  xxxxx                    active    
1002 fddi-default                     act/unsup 
1003 token-ring-default               act/unsup 
1004 fddinet-default                  act/unsup 
1005 trnet-default                    act/unsup 

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0   
2    enet  100002     1500  -      -      -        -    -        0      0   
3    enet  100003     1500  -      -      -        -    -        0      0   
4    enet  100004     1500  -      -      -        -    -        0      0   
10   enet  100010     1500  -      -      -        -    -        0      0   
11   enet  100011     1500  -      -      -        -    -        0      0   
20   enet  100020     1500  -      -      -        -    -        0      0   
38   enet  100038     1500  -      -      -        -    -        0      0   
39   enet  100039     1500  -      -      -        -    -        0      0   
60   enet  100060     1500  -      -      -        -    -        0      0   
62   enet  100062     1500  -      -      -        -    -        0      0   
63   enet  100063     1500  -      -      -        -    -        0      0   
100  enet  100100     1500  -      -      -        -    -        0      0   
900  enet  100900     1500  -      -      -        -    -        0      0   
1002 fddi  101002     1500  -      -      -        -    -        0      0   
1003 tr    101003     1500  -      -      -        -    -        0      0   
1004 fdnet 101004     1500  -      -      -        ieee -        0      0   
1005 trnet 101005     1500  -      -      -        ibm  -        0      0   

Remote SPAN VLANs
------------------------------------------------------------------------------
900

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

SWT02#sh vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Gi1/0/26, Gi1/0/28
2    xxx                           active    
3    xxx                     active    
4    xxx                  active    
10   xxx              active    Gi1/0/12
11   xxx               active    Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/5, Gi1/0/6, Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Gi1/0/17
                                                Gi1/0/18
20   xxx              active    
38   xxx              active    
39   xxx              active    
60   xxx              active    
62   xxx                     active    
63   xxx                     active    
100  xxx             active    
900  xxx                   active    
1002 fddi-default                     act/unsup 
1003 token-ring-default               act/unsup 
1004 fddinet-default                  act/unsup 
1005 trnet-default                    act/unsup 

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0   
2    enet  100002     1500  -      -      -        -    -        0      0   
3    enet  100003     1500  -      -      -        -    -        0      0   
4    enet  100004     1500  -      -      -        -    -        0      0   
10   enet  100010     1500  -      -      -        -    -        0      0   
11   enet  100011     1500  -      -      -        -    -        0      0   
20   enet  100020     1500  -      -      -        -    -        0      0   
38   enet  100038     1500  -      -      -        -    -        0      0   
39   enet  100039     1500  -      -      -        -    -        0      0   
60   enet  100060     1500  -      -      -        -    -        0      0   
62   enet  100062     1500  -      -      -        -    -        0      0   
63   enet  100063     1500  -      -      -        -    -        0      0   
100  enet  100100     1500  -      -      -        -    -        0      0   
900  enet  100900     1500  -      -      -        -    -        0      0   
1002 fddi  101002     1500  -      -      -        -    -        0      0   
1003 tr    101003     1500  -      -      -        -    srb      0      0   
1004 fdnet 101004     1500  -      -      -        ieee -        0      0   
1005 trnet 101005     1500  -      -      -        ibm  -        0      0   

Remote SPAN VLANs
------------------------------------------------------------------------------
900

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

Leo Laohoo · ‎02-23-2019

@Peter Nesta wrote:

001687: Feb 22 16:03:06.467 GMT: %UDLD-4-UDLD_PORT_DISABLED: UDLD disabled interface Gi1/0/27, neighbor mismatch detected
001688: Feb 22 16:03:06.467 GMT: %PM-4-ERR_DISABLE: udld error detected on Gi1/0/27, putting Gi1/0/27 in err-disable state

The issue is the other side of this switch.

Another thing: Input errors are RECEIVED. This means that the other side of Gi1/0/27 is sending bad frames.

Check the interface of the remote side of Gi 1/0/27 and there should be no CRC errors.

Peter Nesta · ‎02-23-2019

Thats the point also the Interfaces on the other Switches have only input errors and CRC error. So both have input and crc errors.

balaji.bandi · ‎02-23-2019

Since you have described all the test you have done both the side SFP, fibre and other test.

I can advise at this stage is, you can conduct p2p interface between switch with single link, rather po and see if the link come back and normal operation ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Leo Laohoo · ‎02-23-2019

@Peter Nesta wrote:

Thats the point also the Interfaces on the other Switches have only input errors and CRC error. So both have input and crc errors.

If both ends are having CRC errors then the horizontal cabling is faulty.