Re: CRC and other port errors to syslog

Tboss99 · ‎03-12-2020

Hi,

We have a bunch of 2911 and 4331 (and a few 2800 left) routers in branch offices around the world.

Is there a way to get these routers to log when port errors occur? Like any input/output interface errors, especially CRC/FCS?

Or is the only option to poll this info with snmp (or cli) ?

Georg Pauwen · ‎03-12-2020

Hello,

you could run a simple EEM script. The script below would run every 60 seconds and generate a syslog message for CRC, input, and output errors for all Ethernet interfaces. It also includes an action to clear the log after 30 seconds, in order to avoid filling up the buffers:

event manager applet SYSLOG
event timer watchdog time 60
action 1 cli command "enable"
action 2 cli command "sh interfaces | include Ethernet|CRC|input errors|output errors"
action 3 syslog msg "$_cli_result"

action 4.0 wait 30

action 5.0 cli command "clear log" pattern "confirm"

action 6.0 cli command "y"

action 7.0 cli command "end"

Mohammad Ramadan A.Hafiez · ‎05-25-2022

Hello Georg,

Thanx for this script, I intend to use it, but when I tested it the output was just

May 25 11:09:34: %HA_EM-6-LOG: SYSLOG: EFTA-2960x-EDGE30#
May 25 11:10:33: %HA_EM-6-LOG: SYSLOG: EFTA-2960x-EDGE30#
May 25 11:11:34: %HA_EM-6-LOG: SYSLOG: EFTA-2960x-EDGE30#
May 25 11:12:33: %HA_EM-6-LOG: SYSLOG: EFTA-2960x-EDGE30#

I think this log occures each time an interface error ok, correct?

also as you can see, the interface number has not been mentioned, how to add it please?

switch applied to c2960x and 2960s, ios-xe 9200

Dan Frey · ‎05-25-2022

Attached is an EEM TCL policy that I believe will do what you want. The policy runs every 5 minutes (or can run manually) and captures input and output errors and CRC errors and compares them to the previous counter (last time the script ran 5 minutes ago). If the counters increment then send a syslog. The first time the script runs there are no previous counters so it will not send any syslog. The syslog messages look like this and it runs this for every interface on the platform that has the error counters.

*May 25 22:50:00.966: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet1 has 20656 input errors (35 input errors this iteration)
*May 25 22:50:01.593: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet3 has 2019 input errors (4 input errors this iteration)
*May 25 22:50:01.906: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet4 has 18281 input errors (40 input errors this iteration)
*May 25 22:55:00.963: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet1 has 20758 input errors (102 input errors this iteration)
*May 25 22:55:01.588: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet3 has 2029 input errors (10 input errors this iteration)
*May 25 22:55:01.902: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet4 has 18370 input errors (89 input errors this iteration)
*May 25 23:00:00.964: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet1 has 20861 input errors (103 input errors this iteration)
*May 25 23:00:01.489: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet3 has 2039 input errors (10 input errors this iteration)
*May 25 23:00:01.803: %HA_EM-6-LOG: monitor_interface_errors.tcl: Interface GigabitEthernet4 has 18460 input errors (90 input errors this iteration)

The instructions to load the tcl file are in the comments section of the script.

- Dan

Isaac Goscha · ‎03-25-2024

Greetings Dan,

Firstly, thank you for putting in the time to make this script. I was having a slight issue with this when trying to implement it. I was getting the following error on the script's second run, so every 10 minutes:

Mar 25 14:18:12.234: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: can't read "inputerror": no such variable
Mar 25 14:18:12.234: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: while executing
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: "if {$inputerror > $previouscntrIE} {
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: set diff [expr $inputerror - $previouscntrIE]
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: action_syslog msg "Interface $intf ..."
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: ("foreach" body line 20)
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: invoked from within
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: "foreach line $lines {
Mar 25 14:18:12.235: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: if [regexp {(^[A-Za-z0-9\/]+)} $line match intf] {
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: if [catch {cli_exec $cli1(fd) "show interfaces $intf | inc protoc..."
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: invoked from within
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: "$slave eval $Contents"
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: (procedure "eval_script" line 7)
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: invoked from within
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: "eval_script slave $scriptname"
Mar 25 14:18:12.236: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: invoked from within
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: "if {$security_level == 1} { #untrusted script
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: interp create -safe slave
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: interp share {} stdin slave
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: interp share {} stdout slave
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: ..."
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: (file "tmpsys:/lib/tcl/base.tcl" line 50)
Mar 25 14:18:12.237: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: Tcl policy execute failed:
LAB_SWITCH#
Mar 25 14:18:12.238: %HA_EM-6-LOG: monitor_interface_errors_any_test.tcl: can't read "inputerror": no such variable

This is on our test switch, which does not currently have any errors on any interface, not sure if that info is of any help. The lab switch is a WS-C3850-24T running Cisco IOS XE Software, Version 16.12.07 . Any help would be appreciated.

Our goal is to also have the script shutdown the interface should it show a large increase in CRC errors, but exclude a couple of specific interfaces from being shutdown as they are not redundant. I modified the script slightly with the following if statement inside of the crcerror section:

if {$intf == "InterfaceName1" || $intf == "InterfaceName2" || $intf == "InterfaceName3"} {
action_syslog msg "Interface $intf is excluded from being shutdown, troubleshoot as needed"
} else {
ios_config "interface $intf" "shutdown"
action_syslog msg "Interface $intf has been shutdown due to crc errors"
}

The crcerror statement was also modified:

if {$crcerror > ($previouscntrCRC + 999)} {

The above two modifications were only added after we confirmed the script would not run without the error mentioned at the beginning.

Dan Frey · ‎03-25-2024

variable inputerror is not getting a value and is causing the error.

The script gets the list of all interface names from "show ip int brief | exclude Interface"

Once it has the interface list it iterates over all interfaces and executes.

"show interfaces $intf | inc protocol|CRC|input errors|output errors". This command may not show input errors to parse out the way the other devices show them. I suspect you will need to update regex to account for the change in matching the output of this command for the various interface types on the platform.

Mohammad Ramadan A.Hafiez · ‎05-26-2022

Hello Dan,

Thank you so much Dan for the file, I will apply and feedback you.

thank you so much again.