11-29-2024 03:45 PM
A few days ago, I was contacted by our security department reporting several IP cameras were down. Upon inspection, I discovered 7 ports in an err_disable state. Looking at the logs, I found the following entries:
Nov 26 13:47:11 PST: %PM-4-ERR_DISABLE: psecure-violation error detected on Gi7/0/30, putting Gi7/0/30 in err-disable state
Nov 26 13:47:11 PST: %PORT_SECURITY-2-PSECURE_VIOLATION_MAC_MOVE: Security violation occurred, caused by MAC address 0800.1011.9fe5 on port GigabitEthernet7/0/34 attempting to access port GigabitEthernet7/0/30.
Nov 26 13:47:12 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet7/0/30, changed state to down
I checked interface gi7/0/34 and found two different MAC addresses on the port. One address was the camera connected to that interface. The other was the MAC address listed in the log entry above. I shut/no shut each interface and they all came back up correctly. In checking interface gi7/0/34 following the shutdown, it only lists one MAC address.
any thoughts as to what occurred?
As a side note, all of the cameras are mounted in the ceiling with no easy physical access.
11-29-2024 04:09 PM
This Mac address 0800.1011.9fe5 does not belong to any vendor. Hard to say anything.
Was there any change on the network?
11-29-2024 04:12 PM
No network changes. I also discovered it was an invalid MAC. This is what makes it so mysterious.
11-29-2024 04:14 PM
Is this happening all throughout the network or just one (or a few) switches?
11-29-2024 04:16 PM
Just this one switch, and just this one incident. The switch has been up for about 2 months (a relatively new install replacing a C4506). The cameras have been in place for about 2 years.
11-29-2024 05:13 PM
If this happened to just one switch, please post the complete output to the command "sh version".
I want to see the firmware version, the model of the switch and the uptime.
12-10-2024 10:59 AM
Cisco IOS XE Software, Version 17.12.04
Cisco IOS Software [Dublin], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.12.4, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2024 by Cisco Systems, Inc.
Compiled Tue 23-Jul-24 09:40 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2024 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.9.3r, RELEASE SOFTWARE (P)
RRMC-1SC-S1 uptime is 7 weeks, 36 minutes
Uptime for this control processor is 6 weeks, 6 days, 3 hours, 0 minutes
System returned to ROM by SSO Switchover
System restarted at 08:59:42 PDT Wed Oct 23 2024
System image file is "bootflash:packages.conf"
Last reload reason: EHSA standby down
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
Technology Package License Information:
------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage
Smart Licensing Status: Smart Licensing Using Policy
cisco C9410R (X86) processor (revision V01) with 1826231K/6147K bytes of memory.
Processor board ID FOX2637PDWX
2 Virtual Ethernet interfaces
192 Gigabit Ethernet interfaces
8 TwentyFive Gigabit Ethernet interfaces
8 Hundred Gigabit Ethernet interfaces
144 Five Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
15927500K bytes of physical memory.
20992000K bytes of Bootflash at bootflash:.
1638400K bytes of Crash Files at crashinfo:.
234430023K bytes of SATA hard disk at disk0:.
20992000K bytes of Bootflash at bootflash-1-0:.
1638400K bytes of Crash Files at crashinfo-1-0:.
234430023K bytes of SATA hard disk at disk0-1-0:.
Base Ethernet MAC Address : 08:f3:fb:a8:4f:c0
Motherboard Assembly Number : 4F15
Motherboard Serial Number : FOC26200M02
Model Revision Number : V02
Motherboard Revision Number : 5
Model Number : C9410R
System Serial Number : FOX2637PDWX
11-29-2024 05:16 PM
Any possibility someone disconnected the camera connections and tried connecting another device?
What was the time interval between the various ports going disabled?
As to mystery MAC being from an unknown vendor, possibly it was a LAA MAC.
11-29-2024 09:46 PM
do
show port security address <<- check if the Mac is learn in other port SW different than g7/0/30
also share
show port secuirty interface g7/0/30
MHM
12-10-2024 11:02 AM
RRMC-1SC-S1#sh port-security interface gi7/0/30
Port Security : Enabled
Port Status : Secure-up
Violation Mode : Shutdown
Aging Time : 0 mins
Aging Type : Absolute
SecureStatic Address Aging : Disabled
Maximum MAC Addresses : 2
Total MAC Addresses : 1
Configured MAC Addresses : 0
Sticky MAC Addresses : 0
Last Source Address:Vlan : 0002.d172.4244:114
Security Violation Count : 1
12-10-2024 11:14 AM
Security Violation Count : 1 <<- there is one violation in SW
So Mac is move between
G7/0/30 and G7/0/34
To do more check
Do
Show port-secuirty mac address
Show mac address g7/0/30 & 34
See mac address in port secuirty match that appears in mac table
MHM
12-10-2024 11:21 AM
RRMC-1SC-S1#sh port-security address | in 0800.1011.9fe5
114 0800.1011.9fe5 SecureDynamic Gi7/0/34 -
RRMC-1SC-S1#sh mac add int gi7/0/30
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
114 0002.d172.4244 STATIC Gi7/0/30
Total Mac Addresses for this criterion: 1
RRMC-1SC-S1#sh mac add int gi7/0/34
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
114 0002.d174.8a15 STATIC Gi7/0/34
114 0800.1011.9fe5 STATIC Gi7/0/34
Total Mac Addresses for this criterion: 2
12-10-2024 11:22 AM
Interestingly enough - this happened again yesterday, but only to one interface.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide