3850 switchports connecting to Q-SYS NV-32-H constantly resetting

[nicolas.vallot]

Our AV integrator is installing new Q-SYS audio/video decoder in a large meeting room. They have connected Q-SYS audio/video encoders/decoders (model nv-32-H) to a dedicated 3850 switch connected to the production network. 

This switch has a data and a voice VLAN. Other devices connect fine and are stable, however these controllers keep dropping of the network. In the switch logs I can see the ports going up and down every minute or so.

The devices need PoE++ so we have connected PoE injectors.

On the switch, I have enabled LLDP and as a test disabled spanning-tree on the affected ports, but still no luck.

Nov 24 16:11:30.019: ILP:: ilp enabled in hwidb Gi1/0/3
Nov 24 16:11:30.019: ilpower_hwif_comingup: Gi1/0/3
Nov 24 16:11:30.019: (Gi1/0/3)ILP notify LLDP-TLV: lldp power class tlv:
Nov 24 16:11:30.020: (Gi1/0/3)(curr/prev) pwr value 15400/0
Nov 24 16:11:30.020: ILP:: ILP CLI 'no shut' handling ( Gi1/0/3 ) Okay
Nov 24 16:11:30.020: ilpower_get_admin_state( Gi1/0/3 )
Nov 24 16:11:30.021: ILP:: Sending poe coredump msg to slot:1
Nov 24 16:11:30.021: ILP::
Sending E_ILP_GET_DEBUG_CORE_DUMP IPC message from RP to platform

Nov 24 16:11:30.022: ILP:: ilp hwidb Gi1/0/3 admstate 2
Nov 24 16:11:30.022: ILP:: ilp hwidb Gi1/0/3 admstate auto, start detect 2
Nov 24 16:11:30.023: ILP:: ILP CLI 'no shut' handling ( Gi1/0/3 ) Okay
Nov 24 16:11:30.024: ILP:: ilp enabled in hwidb Gi1/0/3
Nov 24 16:11:30.024: ilpower_if_statechange_complete: Gi1/0/3
Nov 24 16:11:30.028: ilpower_get_admin_state( Gi1/0/3 )
Nov 24 16:11:30.028: ILP:: Sending poe detect msg to slot:1 port:3
Nov 24 16:11:30.029: ILP::
Sending E_ILP_START_IEEE IPC message from RP to platform

Nov 24 16:11:30.119: ILP:: Rx Response ILP msg: response_code 14, sw_num 1
Nov 24 16:11:30.119: ILP:: ILP msg: received E_ILP_GET_DEBUG_CORE_DUMP
Nov 24 16:11:30.119: ILP:: Inline power process coredump for switch 1
Nov 24 16:11:30.162: ILP:: Rx Response ILP msg: response_code 12, sw_num 1
Nov 24 16:11:30.162: ILP:: ILP msg: received E_ILP_GET_POWER_SENSE
Nov 24 16:11:30.162: ILP:: ILP:pwr_sense: num_ports: 24, switch_num: 1
Nov 24 16:11:30.392: ILP:: Rx Response ILP msg: response_code 14, sw_num 1
Nov 24 16:11:30.392: ILP:: ILP msg: received E_ILP_GET_DEBUG_CORE_DUMP
Nov 24 16:11:30.393: ILP:: Inline power process coredump for switch 1
Nov 24 16:11:30.856: ILP:: Rx Response ILP msg: response_code 13, sw_num 1
Nov 24 16:11:30.856: ILP:: ILP msg: received E_ILP_GET_ALL_EVENTS
Nov 24 16:11:30.857: ILP:: ILP:get_all_events: num_ports: 1, sw_num: 1 ieee_rev: 0
Nov 24 16:11:30.857: ILP:: ILP:get_all_events: num_port: 1, if_id: 3
Nov 24 16:11:30.857: ILP:: interface in get_all_events: Gi1/0/3, slot 1, port 3
Nov 24 16:11:30.857: ILP:: ilp event NONE
Nov 24 16:11:30.857: ILP:: ilp fault 11
Nov 24 16:11:30.858: ILP:: posting ilp slot 1 port 3 event 29 class 0
Nov 24 16:11:30.858: ilpower_get_admin_state( Gi1/0/3 )

Nov 24 16:11:32.016: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to down
Nov 24 16:11:33.116: ILP:: ilpsm posting link up event Gi1/0/3
Nov 24 16:11:35.106: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
Nov 24 16:11:35.111: ILP:: ilp enabled in hwidb Gi1/0/3
Nov 24 16:11:35.111: ilpower_if_statechange_complete: Gi1/0/3
Nov 24 16:11:35.111: (Gi1/0/3)ILP notify LLDP-TLV: lldp power class tlv:
Nov 24 16:11:35.112: (Gi1/0/3)(curr/prev) pwr value 15400/0

After that the port goes down.

As a test I disabled PoE on that port (power inline never) but it didn't make any difference.

Has anyone run into similar issues and know how to resolve this ?

Thanks.

[marce1000]

 - Compare current running software version to latest advisory available, it's always advisable to try the latter (then) , if applicable ,  

 M.

[Leo Laohoo]

Do this:

conf t
 interface Gi1/0/3
 power inline never
end
test cable tdr interface Gi1/0/3

Give it about 10 seconds and then post the complete output of the command below: 

show cable tdr interface Gi1/0/3

And do not forget to re-enable PoE to the port: 

conf t
 interface Gi1/0/3
 NO power inline never
end

[nicolas.vallot]

It's a good idea to test the cabling but I don't think that's causing it. Only the 5 Q-SYS devices connected to that switch are having problems. The other equipment has a stable network connection. However, here are the test results. Cabling seems fine.

SWITCH#sh cable tdr int gi1/0/3
TDR test last run on: November 25 13:58:41

Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/3 1000M Pair A 15 +/- 10 meters Pair B Normal
Pair B 15 +/- 10 meters Pair A Normal
Pair C 15 +/- 10 meters Pair D Normal
Pair D 15 +/- 10 meters Pair C Normal

We are running 16.12.9, which is our standard version used accross all our 3850 switches.

[nicolas.vallot]

Same additional checks I have done:

SWITCH#sh int gi1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 2852.6171.1003 (bia 2852.6171.1003)
Description: 
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 00:25:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 24000 bits/sec, 17 packets/sec
5 minute output rate 912000 bits/sec, 80 packets/sec
192695238 packets input, 23891004438 bytes, 0 no buffer
Received 1453713 broadcasts (1394383 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1394383 multicast, 0 pause input
0 input packets with dribble condition detected
196869629 packets output, 27450029058 bytes, 0 underruns
Output 8211 broadcasts (0 multicasts)
0 output errors, 0 collisions, 18 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

CORE_SWITCH#ping vrf GUEST x.x.x.x repeat 5000
Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 172.19.251.21, timeout is 2 seconds:
.......!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!..........!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..........!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

So although the port is flapping every second, I can still ping the device connect to it, however there are some failures, happening on a regular basis. 

From the console logs:

Nov 25 13:10:44.160: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
Nov 25 13:10:45.167: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to down
Nov 25 13:10:48.174: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
Nov 25 13:10:49.174: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
Nov 25 13:11:10.371: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
Nov 25 13:11:11.377: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to down
Nov 25 13:11:14.431: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
Nov 25 13:11:15.431: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up

Could it be caused by multicast flooding ?

[Leo Laohoo]

Are there any connections in Gi 1/0/2 and Gi 1/0/4? 
What is the uptime of this switch?

[nicolas.vallot]

Yes devices (different type, not Q-SYS controller) connected to Gi1/02 and Gi1/0/4 are stable.

It's only these Q-SYS devices that are having this problem.

SWITCH uptime is 1 week, 5 days, 3 hours, 24 minutes

[Leo Laohoo]

Move the Gi 1/0/3 to a port with nothing beside it.  

[marce1000]

 - You may find this useful : https://www.qsys.com/resource-files/productresources/dn/switch_configuration_examples/audio_video/cisco/q_dn_qsys_cisco_catalyst_3650_3850_sce.pdf

 M.