Re: Intermediate Port Shutoff

GloballyUnique · ‎12-03-2025

My inside trunked router port randomly shuts, with no indication of why in the logs. I had debug STP. What's another debug I can enable to try and get an idea why?
Router Log

Dec 3 2025 01:26:37 UTC: %TRACK-6-STATE: 22 interface Gi0/0/1.2 ip routing Up -> Down
Dec 3 2025 01:26:37 UTC: %TRACK-6-STATE: 31 interface Gi0/0/1.3 line-protocol Up -> Down
Dec 3 2025 01:26:37 UTC: %TRACK-6-STATE: 32 interface Gi0/0/1.3 ip routing Up -> Down

Switch Log
Dec 3 2025 01:29:00 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/4, changed state to down

GloballyUnique · ‎12-03-2025

Hardware & Software

PACSTAR 441 IOS XE v17.04.01

PACSTAR 447 IOS XE v17.09.04a

Kasun Bandara · ‎12-03-2025

@GloballyUnique hi, check the #show interface x/x/x

check if there is any, CRC errors, drops , runts , etc

Please rate this and mark as solution/answer, if this resolved your issue
Good luck
KB

GloballyUnique · ‎12-04-2025

Added output's to reply below, I will get the again whenever it shuts down again.

Cristian Matei · ‎12-04-2025

Hi,

How often does it happen? Is it periodically with more or less same time gap between events? Is this a copper or fiber link? Is it a direct link or there are additional passive/active devices in the path (patch-panel, switch, etc)? Can you paste the output of "show interfaces Gi0/0/1" from the router (assuming this is the flapping link) and "show interfaces Gi1/4" from the switch (assuming this is the flapping link) ?

Thanks,

Cristian.

GloballyUnique · ‎12-04-2025

On average it happens around every 4 hours, but sometimes it will go down up to 4 times within an hour. The "burst" are sometimes at night, sometimes in the mornings.

Copper, no extra devices between the router and switch where link occurs.

Here are sh int outputs. It last went down approx 2 hours ago. Added one sh int of the sub-int's. There are 7 sub interfaces.

-------NOT DOWN--------
switch#sh int g1/4
GigabitEthernet1/4 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is ac3a.67cb.4e64 (bia ac3a.67cb.4e64)
Description: TRUNK TO ROUTER01 OF RINTER VLAN ROUTING
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:35, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 88000 bits/sec, 15 packets/sec
5 minute output rate 387000 bits/sec, 51 packets/sec
4956020 packets input, 1847996689 bytes, 0 no buffer
Received 1636638 broadcasts (1555509 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1555509 multicast, 0 pause input
0 input packets with dribble condition detected
10885080 packets output, 6287841447 bytes, 0 underruns
Output 8234379 broadcasts (0 multicasts)
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

router#sh int g0/0/1
GigabitEthernet0/0/1 is up, line protocol is up
Hardware is ESR-6300-2x1GE, address is 549f.c67c.7c01 (bia 549f.c67c.7c01)
Description: (GigabitEthernet0/0/1) TRUNK FOR INTER VLAN ROUTING
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is force-up, media type is RJ45
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 2164
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 364000 bits/sec, 53 packets/sec
5 minute output rate 72000 bits/sec, 15 packets/sec
6890275 packets input, 4567259018 bytes, 0 no buffer
Received 457577 broadcasts (3035178 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4728569 multicast, 0 pause input
3130029 packets output, 1178066292 bytes, 0 underruns
Output 50012 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 15 interface resets
4534 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
17 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
SFFrouter#
SFFrouter#sh int g0/0/1.3
GigabitEthernet0/0/1.3 is up, line protocol is up
Hardware is ESR-6300-2x1GE, address is 549f.c67c.7c01 (bia 549f.c67c.7c01)
Description: GigabitEthernet0/0/1.3 DATA VLAN Sub-Interface
Internet address is 192.168.120.2/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 3.
ARP type: ARPA, ARP Timeout 04:00:00
Keepalive not supported
Last clearing of "show interface" counters never

GloballyUnique · ‎12-04-2025

Here is a snapshot of the int g0/0/1 right after it went down. Also, unplugging and replugging the cable alone do not bring it back up. It has to be shut then no shut. I also got a output from debugging Platform.

Dec 5 2025 01:53:27 UTC: %IOSXE-6-PLATFORM: R0/0: kernel: mvpp2x f2000000.mvpp2x_ethernet mvpp2: link down

GigabitEthernet0/0/1 is down, line protocol is down
Hardware is ESR-6300-2x1GE, address is 549f.c67c.7c01 (bia 549f.c67c.7c01)
Description: (GigabitEthernet0/0/1) TRUNK FOR INTER VLAN ROUTING
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is force-up, media type is RJ45
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:05:12, output 00:05:12, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 2164
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
7743189 packets input, 5171321143 bytes, 0 no buffer
Received 507386 broadcasts (3446393 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5344412 multicast, 0 pause input
3487714 packets output, 1312498917 bytes, 0 underruns
Output 55987 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 16 interface resets
5078 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
19 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Cristian Matei · ‎12-08-2025

Hi,

Can you paste the configuration of the port, as well as the confguration of the port on the remote side? Also, can you paste the output of following commands from the router in question: "show version", "show platform diag", "show environment", "show environment all", "show platform software status control-processor"?

Do you have the luxury of performing an IOS reload and/or upgrade?

Thanks,

Cristian.

GloballyUnique · ‎12-08-2025

We could potentially due a IOS upgrade/ reload but I would have to run that by management. Below are the show version, show plaform diag, show environment, show environment all, and show platform software status control-processor.

--------Router Flapping Port Config (Running config)--------

interface GigabitEthernet0/0/1
description (GigabitEthernet0/0/1) TRUNK FOR INTER VLAN ROUTING
no ip address
media-type rj45
speed 1000
no negotiation auto
vlan-id dot1q 4094
!
spanning-tree bpduguard disable

interface GigabitEthernet0/0/1.2
description GigabitEthernet0/0/1.2 MGT VLAN Sub-Interface
encapsulation dot1Q 2
ip address 172.17.0.2 255.255.0.0
no ip proxy-arp
zone-member security ZONE_MGT
standby version 2
standby 2 ip 172.17.0.1
standby 2 timers 1 3
standby 2 priority 254
standby 2 preempt delay minimum 30
standby 2 authentication md5 key-chain HSRPCAC2S
standby 2 name HSRP2
standby 2 track 11 decrement 20

interface GigabitEthernet0/0/1.3
description GigabitEthernet0/0/1.3 DATA VLAN Sub-Interface
encapsulation dot1Q 3
ip address 192.168.120.2 255.255.255.0
ip helper-address 192.168.36.18
no ip proxy-arp
ip pim neighbor-filter PIM_INTERNAL_NEIGHBORS
ip pim sparse-dense-mode
ip nat inside
zone-member security ZONE_INTERNAL
standby version 2
standby 3 ip 192.168.120.1
standby 3 timers 1 3
standby 3 priority 254
standby 3 preempt delay minimum 30
standby 3 authentication md5 key-chain HSRPCAC2S
standby 3 name HSRP3
standby 3 track 11 decrement 20
ntp broadcast

interface GigabitEthernet0/0/1.5
description GigabitEthernet0/0/1.5 DMZ VLAN Sub-Interface
encapsulation dot1Q 5
ip address 10.10.10.12 255.255.255.248
no ip proxy-arp
ip pim neighbor-filter PIM_INTERNAL_NEIGHBORS
ip pim dr-priority 254
ip pim sparse-dense-mode
zone-member security ZONE_DMZ
standby version 2
standby 5 ip 10.10.10.11
standby 5 timers 1 3
standby 5 priority 254
standby 5 preempt delay minimum 30
standby 5 authentication md5 key-chain HSRPCAC2S
standby 5 name HSRP5
standby 5 track 11 decrement 20

interface GigabitEthernet0/0/1.6
description GigabitEthernet0/0/1.6 VOICE VLAN Sub-Interface
encapsulation dot1Q 6
ip address 172.18.0.2 255.255.0.0
ip helper-address 192.168.36.18
no ip proxy-arp
ip pim neighbor-filter PIM_INTERNAL_NEIGHBORS
ip pim dr-priority 254
ip pim sparse-dense-mode
zone-member security ZONE_VOICE
standby version 2
standby 6 ip 172.18.0.1
standby 6 timers 1 3
standby 6 priority 254
standby 6 preempt delay minimum 30
standby 6 authentication md5 key-chain HSRPCAC2S
standby 6 name HSRP6
standby 6 track 11 decrement 20

interface GigabitEthernet0/0/1.7
description GigabitEthernet0/0/1.7 VM Management VLAN Sub-Interface
encapsulation dot1Q 7
ip address 192.168.7.2 255.255.255.0
no ip proxy-arp
zone-member security ZONE_MGT
standby version 2
standby 7 ip 192.168.7.1
standby 7 timers 1 3
standby 7 priority 254
standby 7 preempt delay minimum 30
standby 7 authentication md5 key-chain HSRPCAC2S
standby 7 name HSRP7
standby 7 track 11 decrement 20

interface GigabitEthernet0/0/1.8

interface GigabitEthernet0/0/1.20
description GigabitEthernet0/0/1.20 TUNNELS VLAN Sub-Interface
encapsulation dot1Q 20
ip address 172.20.0.2 255.255.0.0
no ip proxy-arp
zone-member security ZONE_TUNNEL

interface GigabitEthernet0/0/1.36
description GigabitEthernet0/0/1.36 ADMIN VLAN Sub-Interface
encapsulation dot1Q 36
ip address 192.168.36.2 255.255.255.0
no ip proxy-arp
ip pim neighbor-filter PIM_INTERNAL_NEIGHBORS
ip pim dr-priority 254
ip pim sparse-dense-mode
ip nat inside
zone-member security ZONE_ADMIN
standby version 2
standby 36 ip 192.168.36.1
standby 36 timers 1 3
standby 36 priority 254
standby 36 preempt delay minimum 30
standby 36 authentication md5 key-chain HSRPCAC2S
standby 36 name HSRP36
standby 36 track 11 decrement 20

interface GigabitEthernet0/0/1.91
description GigabitEthernet0/0/1.91 SPARE ROUTER TO ROUTER INTERFACE
encapsulation dot1Q 91
ip address 192.168.91.1 255.255.255.248
no ip proxy-arp

interface GigabitEthernet0/0/1.92
description GigabitEthernet0/0/1.92 ROUTER TO ROUTER INTERFACE FOR EXCHANGING EIGRP ROUTES
encapsulation dot1Q 92
ip address 192.168.92.1 255.255.255.248
no ip proxy-arp
ip authentication mode eigrp 101 md5
ip authentication key-chain eigrp 101 EIGRP-CAC2S
zone-member security ZONE_INTERNAL
distribute-list OUTBOUND_ROUTE_FILTER out GigabitEthernet0/0/0
distribute-list INBOUND_ROUTE_FILTER in GigabitEthernet0/0/0
offset-list EIGRP_DMZ_OFFSET_LIST out 0 GigabitEthernet0/0/0
no passive-interface GigabitEthernet0/0/0
no passive-interface GigabitEthernet0/0/1.92
distribute-list OUTBOUND_ROUTE_FILTER out GigabitEthernet0/0/0
distribute-list INBOUND_ROUTE_FILTER in GigabitEthernet0/0/0
no passive-interface GigabitEthernet0/0/1.20

--------Router "Show Version Output" (Dropped Cisco canner for readability)--------

Cisco IOS XE Software, Version 17.09.04a
Cisco IOS Software [Cupertino], ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9_IOT-M), Version 17.9.4a, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Fri 20-Oct-23 10:57 by mcpre

ROM: 4.1(REL)

SFFrouter uptime is 5 days, 16 hours, 33 minutes
Uptime for this control processor is 5 days, 16 hours, 35 minutes
System returned to ROM by reload at 02:17:05 UTC Wed Dec 3 2025
System restarted at 02:20:49 UTC Wed Dec 3 2025
System image file is "bootflash:c6300-universalk9.17.09.04a.SPA.bin"
Last reload reason: Reload Command

Technology Package License Information:

-----------------------------------------------------------------
Technology Type Technology-package Technology-package
Current Next Reboot
-----------------------------------------------------------------
Smart License Perpetual None None
Smart License Subscription None None

The current throughput level is 50000 kbps

Smart Licensing Status: Smart Licensing Using Policy

cisco ESR-6300-NCP-K9 (1RU) processor with 1354371K/6147K bytes of memory.
Processor board ID FOC24461WA7
Router operating mode: Autonomous
MCU bootloader version: 8
MCU application version: 10
1 Virtual Ethernet interface
6 Gigabit Ethernet interfaces
1 Serial interface
1 terminal line
32768K bytes of non-volatile configuration memory.
3987512K bytes of physical memory.
2887679K bytes of Bootflash at bootflash:.

Configuration register is 0x2102

--------Router "Show Platform Diagnostics"--------

Chassis type: ESR-6300-NCP-K9

Slot: 0, ESR-6300-NCP-K9
Running state : ok
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:47 (5d16h ago)
Software declared up time : 00:01:46 (5d16h ago)
CPLD version :
Firmware version : 4.1(REL)

Sub-slot: 0/0, ESR-6300-2x1GE
Operational status : ok
Internal state : inserted
Physical insert detect time : 00:02:43 (5d16h ago)
Logical insert detect time : 00:02:43 (5d16h ago)

Sub-slot: 0/1, ESR-6300-ES-4
Operational status : ok
Internal state : inserted
Physical insert detect time : 00:02:44 (5d16h ago)
Logical insert detect time : 00:02:44 (5d16h ago)

Slot: R0, ESR-6300-NCP-K9
Running state : ok, active
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:47 (5d16h ago)
Software declared up time : 00:00:47 (5d16h ago)
CPLD version : 00000000
Firmware version : 4.1(REL)

Slot: F0, ESR-6300-NCP-K9
Running state : ok, active
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:47 (5d16h ago)
Software declared up time : 00:02:05 (5d16h ago)
Hardware ready signal time : 00:00:00 (never ago)
Packet ready signal time : 00:02:08 (5d16h ago)
CPLD version : 00000000
Firmware version : 4.1(REL)

Slot: P0, PWR-12V
State : ok
Physical insert detect time : 00:01:38 (5d16h ago)

Slot: GE-POE, Unknown
State : NA
Physical insert detect time : 00:00:00 (never ago)

--------Router Show Enviroment--------

Number of Critical alarms: 0
Number of Major alarms: 0
Number of Minor alarms: 0

Slot Sensor Current State Reading Threshold(Minor,Major,Critical,Shutdown)
---------- -------------- --------------- ------------ ---------------------------------------
R0 Temp: LM75BXXX Normal 39 Celsius (80 ,90 ,96 ,na )(Celsius)

--------Router "Show enviroment all"--------
Sensor List: Environmental Monitoring
Sensor Location State Reading
Temp: LM75BXXX R0 Normal 40 Celsius

--------Router "sh plat software status control-processor"--------

RP0: online, statistics updated 1 seconds ago
Load Average: healthy
1-Min: 0.71, status: healthy, under 5.00
5-Min: 0.89, status: healthy, under 5.00
15-Min: 1.01, status: healthy, under 5.00
Memory (kb): healthy
Total: 3987512
Used: 2761204 (69%), status: healthy
Free: 1226308 (31%)
Committed: 2534640 (64%), under 90%
Per-core Statistics
CPU0: CPU Utilization (percentage of time spent)
User: 1.60, System: 1.60, Nice: 0.00, Idle: 95.70
IRQ: 0.80, SIRQ: 0.30, IOwait: 0.00
CPU1: CPU Utilization (percentage of time spent)
User: 1.40, System: 1.40, Nice: 0.00, Idle: 96.10
IRQ: 0.80, SIRQ: 0.30, IOwait: 0.00
CPU2: CPU Utilization (percentage of time spent)
User: 35.50, System: 1.50, Nice: 0.00, Idle: 62.20
IRQ: 0.70, SIRQ: 0.10, IOwait: 0.00
CPU3: CPU Utilization (percentage of time spent)
User: 24.22, System: 7.60, Nice: 0.00, Idle: 62.16
IRQ: 5.80, SIRQ: 0.20, IOwait: 0.00

--------Switch Config (connected to g0/0/1 on router)--------
interface GigabitEthernet1/4
description TRUNK TO ROUTER01 OF RINTER VLAN ROUTING
switchport trunk native vlan 95
switchport trunk allowed vlan 2-11,16,19-21,31-33,36,41,59,80,90-92,101
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
spanning-tree bpduguard disable
spanning-tree guard root
ip dhcp snooping trust

Cristian Matei · ‎12-08-2025

Hi,

I don't see anything that could justify the interface resets, and the fact that you see a kernel message related to link going down worries me.

First try to change the copper cable, ensure it's not hand made, rather machine made. If still not fixed, perform upgrade and reload the box, I suggest upgrading to 17.12.6 or 17.15.4c.

Thanks,

Cristian.

Dan Frey · ‎12-08-2025

There are quite a few output drops on the interface when it went down (Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 2164). This could be causing the TRACK utility to bring down the line-protocol. The interface is using FIFO so the control plane traffic may not be getting through when the trunk is oversubscribed. Lower speed interfaces we used to change from FIFO to WFQ to fix this. A 1Gbps interface the recommendation is CBWFQ to prioritize control plane traffic during times of congestion.

GloballyUnique · ‎12-08-2025

Adding "debug track all" to my router.

Will look implementing QoS, specifically WFQ.

Thank you for your help!

ethanncollins93 · ‎12-07-2025

It definitely looks like the interface is dropping without a clear trigger, so adding more focused debugging can help narrow things down. Alongside STP debugging, try enabling debugs for interface status, link negotiation, and Ethernet events — these often reveal flaps caused by physical issues, duplex mismatches, or LACP problems. Also check for err-disable causes, speed/duplex mismatches, and any track or SLA dependencies that could be forcing the port down. If the shutdown is truly random, swapping the cable or testing the switch port is also worth doing just to rule out a physical or hardware-level fault.

Thomasanderson33 · ‎12-08-2025

It looks like your trunked router port is flapping, but STP debug alone may not reveal the full cause. You might try enabling debug interface or debug adjacency to see if there are link or protocol-level issues. Also, check for physical layer problems like cable or SFP issues, and ensure both ends of the trunk have matching configurations for speed, duplex, and VLANs.