I'm assuming many others out there run MACsec on the 3560-CX, however I'm totally frustrated and unable to get a stable link.
Between two identical units, the link establishes just fine and CTS/MACsec appear to be working. If I `show cts` `show macsec` etc I get expected output.
Problem is, the link stops passing traffic randomly, for approx 3 minutes at a time, before re-establishing itself. This isn't very often, it happens once every 4-12 hours, but when it occurs, all traffic between sites stops dead.
Notes:
- There is nothing in the log (what debug should I turn on, CTS debug overflows the log very quickly)
- When I configure with two DF links in a port-channel, with LACP, the links go down more frequently (every 1/2 - 3 hours), however the link stops passing traffic for only about 30 seconds (even with lacp fast). I see errors such as below. It isn't the same link that goes down in this scenario either.
Apr 3 05:41:32.014: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to up Apr 3 12:26:03.795: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to down Apr 3 12:26:10.674: %EC-5-L3DONTBNDL2: Te1/0/2 suspended: LACP currently not enabled on the remote port. Apr 3 12:28:28.817: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to up Apr 3 12:54:36.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to down Apr 3 12:54:43.562: %EC-5-L3DONTBNDL2: Te1/0/1 suspended: LACP currently not enabled on the remote port. Apr 3 12:57:01.904: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to up Apr 3 13:40:55.560: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to down Apr 3 13:41:03.359: %EC-5-L3DONTBNDL2: Te1/0/2 suspended: LACP currently not enabled on the remote port. Apr 3 13:43:19.352: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to up Apr 3 18:23:55.728: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/2, changed state to down
3. When the link goes down, I CANNOT ping the 3560CX #1 switch anymore from site A (didn't test the other side). It seems like its not just the link thats down, the whole switch has stopped responding, almost like its regenerating keys or something.
Setup:
2x Cisco Catalyst 3560CX-8XPD-S
2x 10G DAC links to site core switches
2x 10G-Base-BX transcievers running 5KM of single mode fiber (to be encrypted).
The image shows the simplest configuration I could have. (I originally was trying two redundant fiber links in an etherchannel, with similar issues).
interface TenGigabitEthernet1/0/1 switchport trunk allowed vlan 5,7,20,21 switchport trunk pruning vlan 2-4,6,8-19,22-1001 switchport mode trunk cts manual no propagate sgt sap pmk 8765432100000000000000000000000000000000000000000000000000001234 mode-list gcm-encrypt !
interface TenGigabitEthernet1/0/2
switchport trunk allowed vlan 5,7,20,21
switchport trunk pruning vlan 2-4,6,8-19,22-1001
switchport mode trunk
!
Both switches have the above CTS configuration for MACsec on the switch-switch configuration. There isn't much else in this config at all.
