Green / Amber blinking on one side of a Trunk link (Catalyst 9300)

[M81col]

First time posting...

It seems I have an issue between 2 switches that I cannot seem to figure out.  I just recently plugged a new Bosch camera into an IE-4000 switch, the access port LED blinks green.  The camera seems to work fine locally.  I can open the webpage, view the live video, etc.  The trunk link LED rapidly blinks green.

However, on the other side of the trunk (Catalyst 9300 switch) I have some kind of fault on the trunk port.  It will rapidly blink green for about 10seconds and then amber for a second and then back to green again.  If I unplug the camera from the IE-4000 the fault on the Catalyst 9300 seems to go away, it goes back to steady green flashing.

I looked at the error counters on the Catalyst 9300 switch and the input errors are accumulating.  I've also looked at the syslog messages and I  don't see anything for the port on the Catalyst 9300.

[marce1000]

 - In what context are you mentioning trunk links and or the need for , as the camera will be sufficient with host mode port settings, also check the native logs with show logging on the 9300 and the IE400 (?)

 M.

[M81col]

The access layer switch (IE-4000) has a few additional cameras connected to it as well as some other things.  There are a few different Vlans on the IE-4000.  From what I can tell the other cameras are working normally, same model of camera.  I have checked the logs using show logging on both the IE-4000 and the 9300.  I do not see anything pertaining to this issue.  I have logging set to Debugging.

Layout:

Camera ---> POE Injector ---> IE-4000 --->Fiber Trunk link to the 9300 

[Leo Laohoo]

Post the complete output to the command "sh interface <PORT>" of the port in question.

[M81col]

This is the output for both sides of the trunk link.

Catalyst 9300

SW-STACK#show interface g1/0/1   
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 3c8b.7f48.e701 (bia 3c8b.7f48.e701)
Description: Cisco Network Trunk
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:14, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:07:02
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 12273000 bits/sec, 1263 packets/sec
5 minute output rate 269000 bits/sec, 323 packets/sec
111461005 packets input, 145175166934 bytes, 0 no buffer
Received 25275 broadcasts (24743 multicasts)
0 runts, 15894 giants, 0 throttles
15894 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 24743 multicast, 0 pause input
0 input packets with dribble condition detected
28211760 packets output, 2917810710 bytes, 0 underruns
Output 73760 broadcasts (329336 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

IE-4000

SW-FDB#show interface g1/1
GigabitEthernet1/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 6c4e.f630.0281 (bia 6c4e.f630.0281)
Description: Cisco TrustSec link to Network Trunk
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 239000 bits/sec, 400 packets/sec
5 minute output rate 15469000 bits/sec, 1561 packets/sec
177718849 packets input, 13012935731 bytes, 0 no buffer
Received 2203985 broadcasts (1793595 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1793595 multicast, 0 pause input
0 input packets with dribble condition detected
677141588 packets output, 876508348882 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

---

@M81col wrote:
15894 input errors

---

The interface counters on the 9300 was cleared 17 hours ago and the input errors are incrementing.  That's not a good sign.  

Post the complete output to the command "sh controll e Gi 1/0/1".

[M81col]

SW-STACK#show controll e g1/0/1
Transmit GigabitEthernet1/0/1 Receive
320373517726 Total bytes 24495332792857 Total bytes
2975388442 Unicast frames 18987495990 Unicast frames
307905980041 Unicast bytes 24494248624353 Unicast bytes
49715166 Multicast frames 4941550 Multicast frames
8921844234 Multicast bytes 1075525493 Multicast bytes
34568491 Broadcast frames 76453 Broadcast frames
3545693451 Broadcast bytes 8643011 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
0 Minimum size frames 0 Minimum size frames
2995567115 65 to 127 byte frames 623225608 65 to 127 byte frames
40645138 128 to 255 byte frames 1717911577 128 to 255 byte frames
15006942 256 to 511 byte frames 363809288 256 to 511 byte frames
7349174 512 to 1023 byte frames 691183727 512 to 1023 byte frames
309270 1024 to 1518 byte frames 4638336579 1024 to 1518 byte frames
794460 1519 to 2047 byte frames 10958398223 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 3 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 351006 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 3624 msecs AGO

[Leo Laohoo]

---

@M81col wrote:
351006 ValidOverSize frames

---

I've never seen something like this before.  

What global MTU is configured in the remote switch of Gi 1/0/1?

[M81col]

Something like what, could you please clarify?

[M81col]

SW-STACK#show system mtu
Global Ethernet MTU is 1500 bytes.

[Georg Pauwen]

Hello,

from what you are saying, it appears that the trunk link has problems with negotiation. The Bosch camera needs PoE I think, but that should not be a problem. Is the access Vlan you have configured for the Bosch camera allowed (on both ends of the trunk) ?

[M81col]

There are other cameras on this IE-4000 that do not throw the trunk link into an error.  It is only when I connect this new camera to the IE-4000.  The cameras are getting their power from a POE injector.  I have tried Toggling the access port on and off.  Toggling the Trunk link on and off.  Trying the access port that the camera is connected to with Port-Sec and without Port-Sec.

I did notice that the 9300 side is manually set to Duplex-Full Speed-1000.

The IE-4000 side is set to Duplex-Auto Speed-Auto.

I have tried manually setting the IE-4000 duplex and speed to match the 9300 but it will not let me.

[M81col]

**Update**     Layout:   Camera---->POE Injector---->IE-4000---Fiber Trunk--->Catalyst 9300---->Camera workstation

I can ping the camera from the camera workstation.  No amber lights on the trunk port, only steady flashing green.

I can open the camera webpage on the camera workstation.  Trunk port on the 9300 flashes green for about 5 seconds then amber and then back to green for about 5 seconds.  Even though I can open the webpage, the live video is not viewable (other cameras on the IE-4000 are working fine).

I know the camera is working fine because if I connect my laptop to the IE-4000 I can pull up the camera webpage and view the live video with no problem.

The trunk is "Trunking" on both sides.  Both sides have the correct Vlans.  I'm completely stumped......