Catalyst 3650 to 3560CX MACsec switch-to-switch link showing CRC error

andrew.butterworth · ‎11-23-2024

I've been doing some testing with a Catalyst 3650-48PS switch with some IBNS 2.0 configs. The 3650 is connected on a single 1Gbps copper connection to a Catalyst 3560CX with the link configured as L3 (no switchport) and OSPF/OSPFv3. The link is also configured with MACsec. Its working, however I noticed on the 3650 that the interface LED kept going orange. On further inspection, there are CRC errors incrementing on the 3650 interface, but not on the 3560CX side.

The config is really simple and its the same configuration on both switches:

cts manual
  no propagate sgt
  sap pmk 0000000000000000000000000000000000000000000000000000001234ABCDEF mode-list gcm-encrypt
 no cts role-based enforcement

I thought it was the cabling, however I replaced that and still the same. I removed the MACsec configuration and the errors are completely gone and no more orange flashes on the 3650 interface LED.

Both switches are running the latest code (3650 IOS-XE 16.12.12, 3560CX 15.2(7)E11). NTP is synced on both switches (not sure if that makes a difference?). System MTU is set to 9100 on both switches ("system mtu jumbo 9100" on the 3560CX), IP & IPv6 MTU on the link is set to 1500.

Any idea's or is this aesthetic? I ran 10000 pings between the two switches a few times and there are no losses.

marce1000 · ‎11-23-2024

- Check logs on both devices and also configure logging trap informational ; and use a syslog server
after doing that to capture the logs 'from everyone'

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

balaji.bandi · ‎11-23-2024

Can you provide what kind of interface used to connect these devices ?

can you post interface config along with show interface x/x

other side may be thinking could be bug.

If you are using 1G SFP modules for inter switch connection, change system MTU to 1550 byte to ensure support of MACsec overhead.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960cx_3650cx/software/release/15-2_7_e/configuration_guide/b_1527e_consolidated_3560cx_2960cx_cg/m_1522e_sec_macsec_encrypt_3750x_3560x_cg.html

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

andrew.butterworth · ‎11-25-2024

This is a 1Gbps copper interface. Gig1/0/24 on the Cat 3650 and Gig1/0/9 on the Cat 3560CX. The command 'system mtu 9100' was added to the 3650 and the commands 'system mtu 1998' and 'system mtu jumbo 9100' were added to the 3560CX. The interfaces are configured as routed ports (no switchport), on the 3650 side the commands 'ip mtu 1500' and 'ipv6 mtu 1500' were added, there is no need to do this on the 3560CX side as the IP & IPv6 MTU's are 1500 by default.

!! Cat 3650 !!
!
interface GigabitEthernet1/0/24
 no switchport
 ip address x.x.x.x 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip mtu 1500
 ip pim sparse-mode
 ip ospf network point-to-point
 no logging event link-status
 ipv6 address x:x::x/126
 ipv6 enable
 ipv6 mtu 1500
 ipv6 nd other-config-flag
 no ipv6 redirects
 ipv6 ospf 10 area 0
 ipv6 ospf network point-to-point
 cts manual
  no propagate sgt
  sap pmk 0000000000000000000000000000000000000000000000000000001234ABCDEF mode-list gcm-encrypt
 no cts role-based enforcement

!! Cat 3560CX !!
!
interface GigabitEthernet1/0/9
 no switchport
 ip address x.x.x.x 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip pim sparse-mode
 ip ospf network point-to-point
 no logging event link-status
 srr-queue bandwidth share 1 70 25 5
 srr-queue bandwidth shape  3 0 0 0
 priority-queue out
 ipv6 address x:x::x/126
 ipv6 enable
 ipv6 nd other-config-flag
 no ipv6 redirects
 ipv6 ospf 10 area 0
 ipv6 ospf network point-to-point
 mls qos trust dscp
 cts manual
  no propagate sgt
  sap pmk 0000000000000000000000000000000000000000000000000000001234ABCDEF mode-list gcm-encrypt
 no cts role-based enforcement

On the 3560CX side, there are no errors:

GigabitEthernet1/0/9 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is a4b2.396c.9342 (bia a4b2.396c.9342)
  Internet address is 192.168.250.130/30
  MTU 9100 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:04, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 33000 bits/sec, 12 packets/sec
     1676317 packets input, 1227991237 bytes, 0 no buffer
     Received 148153 broadcasts (4451 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 148132 multicast, 0 pause input
     0 input packets with dribble condition detected
     3120052 packets output, 1558034680 bytes, 0 underruns
     0 output errors, 0 collisions, 6 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

On the 3650 side there are constant CRC errors:

GigabitEthernet1/0/24 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 2c5a.0f92.2bc0 (bia 2c5a.0f92.2bc0)
  Internet address is 192.168.250.129/30
  MTU 9100 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 179/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 39000 bits/sec, 2 packets/sec
  5 minute output rate 3000 bits/sec, 2 packets/sec
     1760 packets input, 2739738 bytes, 0 no buffer
     Received 680 broadcasts (58 IP multicasts)
     0 runts, 0 giants, 0 throttles
     6065 input errors, 6065 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 676 multicast, 0 pause input
     0 input packets with dribble condition detected
     2167 packets output, 554766 bytes, 0 underruns
     Output 2 broadcasts (1 IP multicasts)
     0 output errors, 0 collisions, 6 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

If I ping the 3650 from the 3560CX 10000, there are zero losses:

....
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/4/101 ms