04-28-2018 06:59 AM - edited 02-20-2020 09:45 PM
I've been having stability issues with CTS manual MacSec on a pair of 3560-CX switches.
Anyone encountered similar issues, or have suggestions?
With MacSec "cts manual", I have lots of issues including:
rxL2UntaggedPkts = 57 rxL2NotagPkts = 0 rxL2SCMissPkts = 0 rxL2CTRLPkts = 0 rxL3CTRLPkts = 0 rxL3UnknownSAPkts = 0 rxL2BadTagPkts = 0 txL2UntaggedPkts = 0 txL2CtrlPkts = 0 txL3CtrlPkts = 0 txL3UnknownSA = 0 SA Index : 1 rxL2ReplayfailPkts = 210 rxL2AuthfailPkts = 0 rxL2PktsOK = 56586 rxL3AuthCheckFail = 0 rxL3ReplayCheckFail = 0 rxL2SAMissPkts = 0 rxL3EspGcm_Pkts = 0 rxL3InverseCheckfail = 0 txL3Protected = 0 txL2Protected = 2172
#show macsec interface te1/0/1 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Use ES Enable : no Use SCB Enable : no Admin Pt2Pt MAC : forceTrue(1) Pt2Pt MAC Operational : no Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Identifier : Name : ICV length : 16 Data length change supported: yes Max. Rx SA : 8 Max. Tx SA : 8 Max. Rx SC : 4 Max. Tx SC : 4 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : **************** SC state : inUse(1) Elapsed time : 01:22:37 Start time : 7w0d Current AN: 1 Previous AN: 0 Next PN: 912 SA State: inUse(1) Confidentiality : no SAK Unchanged : no SA Create time : 4d15h SA Start time : 7w0d SC Statistics Auth-only Pkts : 0 Auth-only Bytes : 0 Encrypt Pkts : 84723 Encrypt Bytes : 0 SA Statistics Auth-only Pkts : 0 Encrypt Pkts : 3083 Port Statistics Receive Secure Channels SCI : **************** SC state : inUse(1) Elapsed time : 01:22:37 Start time : 7w0d Current AN: 1 Previous AN: 0 Next PN: 971 RX SA Count: 0 SA State: inUse(1) SAK Unchanged : no SA Create time : 4d15h SA Start time : 7w0d SC Statistics Notvalid pkts 0 Invalid pkts 0 Valid pkts 848460 Valid bytes 0 Late pkts 210 Uncheck pkts 0 Delay pkts 14 UnusedSA pkts 0 NousingSA pkts 0 Decrypt bytes 0 SA Statistics Notvalid pkts 0 Invalid pkts 0 Valid pkts 57230 UnusedSA pkts 0 NousingSA pkts 0 Port Statistics
Configuration on both switches:
interface TenGigabitEthernet1/0/1 switchport trunk allowed vlan 5,7,20,21 switchport trunk pruning vlan 2-4,6,8-19,22-1001 switchport mode trunk cts manual no propagate sgt sap pmk ******************************** mode-list gcm-encrypt
#show cts interface te1/0/1 Global Dot1x feature is Disabled Interface TenGigabitEthernet1/0/1: CTS is enabled, mode: MANUAL IFC state: OPEN Interface Active for 01:20:36.962 Authentication Status: NOT APPLICABLE Peer identity: "unknown" Peer's advertised capabilities: "sap" Authorization Status: NOT APPLICABLE SAP Status: SUCCEEDED Version: 2 Configured pairwise ciphers: gcm-encrypt Replay protection: enabled Replay protection mode: STRICT Selected cipher: gcm-encrypt Propagate SGT: Disabled Cache Info: Expiration : N/A Cache applied to link : NONE Critical-Authentication: Disabled Peer SGT: 0 Peer SGT assignment: Untrusted Default PMK: Not Configured Default SGACL: Fail-Open: Enabled Statistics: authc success: 0 authc reject: 0 authc failure: 0 authc no response: 0 authc logoff: 0 sap success: 1 sap fail: 0 authz success: 0 authz fail: 0 port auth fail: 0 L3 IPM: disabled.
04-29-2018 05:35 AM
After a couple of dropped link events, the counters look as follows:
(See the replayfailed and rxL2AuthfailPkts / Notvalid pkts)
#show cts macsec counters interface te1/0/1 CTS Security Statistic Counters: rxL2UntaggedPkts = 960 rxL2NotagPkts = 0 rxL2SCMissPkts = 0 rxL2CTRLPkts = 0 rxL3CTRLPkts = 0 rxL3UnknownSAPkts = 0 rxL2BadTagPkts = 0 txL2UntaggedPkts = 0 txL2CtrlPkts = 0 txL3CtrlPkts = 0 txL3UnknownSA = 0 SA Index : 1 rxL2ReplayfailPkts = 57460 rxL2AuthfailPkts = 777 rxL2PktsOK = 5618455 rxL3AuthCheckFail = 0 rxL3ReplayCheckFail = 0 rxL2SAMissPkts = 0 rxL3EspGcm_Pkts = 0 rxL3InverseCheckfail = 0 txL3Protected = 0 txL2Protected = 4106 GENERIC Counters: CRCAlignErrors = 0 UndersizedPkts = 0 OversizedPkts = 0 FragmentPkts = 0 Jabbers = 0 Collisions = 0 InErrors = 0 OutErrors = 0 ifInDiscards = 0 ifInUnknownProtos = 0 ifOutDiscards = 0 dot1dDelayExceededDiscards = 0 txCRC = 0 linkChange = 0
#show macsec interface te1/0/1 MACsec is enabled Replay protect : enabled Replay window : 0 Include SCI : yes Use ES Enable : no Use SCB Enable : no Admin Pt2Pt MAC : forceTrue(1) Pt2Pt MAC Operational : no Cipher : GCM-AES-128 Confidentiality Offset : 0 Capabilities Identifier : Name : ICV length : 16 Data length change supported: yes Max. Rx SA : 8 Max. Tx SA : 8 Max. Rx SC : 4 Max. Tx SC : 4 Validate Frames : strict PN threshold notification support : Yes Ciphers supported : GCM-AES-128 Transmit Secure Channels SCI : **************** SC state : inUse(1) Elapsed time : 23:56:13 Start time : 7w0d Current AN: 0 Previous AN: 1 Next PN: 1069 SA State: inUse(1) Confidentiality : no SAK Unchanged : no SA Create time : 5d13h SA Start time : 7w0d SC Statistics Auth-only Pkts : 0 Auth-only Bytes : 0 Encrypt Pkts : 6772436 Encrypt Bytes : 0 SA Statistics Auth-only Pkts : 0 Encrypt Pkts : 1068 Port Statistics Receive Secure Channels SCI : **************** SC state : inUse(1) Elapsed time : 23:56:13 Start time : 7w0d Current AN: 0 Previous AN: 1 Next PN: 1015 RX SA Count: 0 SA State: inUse(1) SAK Unchanged : no SA Create time : 5d13h SA Start time : 7w0d SC Statistics Notvalid pkts 92906 Invalid pkts 0 Valid pkts 1878968117 Valid bytes 0 Late pkts 57698 Uncheck pkts 0 Delay pkts 240 UnusedSA pkts 0 NousingSA pkts 0 Decrypt bytes 0 SA Statistics Notvalid pkts 0 Invalid pkts 0 Valid pkts 5257623 UnusedSA pkts 0 NousingSA pkts 0
#show interfaces te1/0/1 TenGigabitEthernet1/0/1 is up, line protocol is up (connected) Hardware is Ten Gigabit Ethernet, address is ******** MTU 1550 bytes, BW 10000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive not set Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LR input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:04, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 21000 bits/sec, 8 packets/sec 5 minute output rate 8000 bits/sec, 11 packets/sec 342320448 packets input, 382844331049 bytes, 0 no buffer Received 1003179 broadcasts (986704 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 986704 multicast, 0 pause input 0 input packets with dribble condition detected 225839613 packets output, 166735018695 bytes, 0 underruns 0 output errors, 0 collisions, 2 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
05-09-2018 07:00 AM
The issue still persists regularly :-/ TAC haven't been much help yet.
When the error occurs, I see the following suspicious bit in the logs:
388338: May 9 17:43:31.495: HULC-MACsec: Invoking the callback for req 1016AA8C 388339: May 9 17:43:31.495: macsec_blocking_callback 388340: May 9 17:43:31.495: Wake up the blocking process 388341: May 9 17:43:31.498: MACsec-CTS: Invoked for SA deletion 388342: May 9 17:43:31.498: MACsec API blocking the invoking context 388343: May 9 17:43:31.502: HULC-MACsec: Entering macsec_req_local_handler req BD0CCEC: 388344: May 9 17:43:31.502: HULC-MACsec: macsec_process_req : Req type: 6 388345: May 9 17:43:31.502: HULC-MACsec: Process delete rxSA requestBD0CCEC for interface TenGigabitEthernet1/0/1 388346: May 9 17:43:31.502: HULC-MACsec: Deleting RxSA with AN = 1Curr AN = 0, Prev AN = 0 388347: May 9 17:43:31.502: Invalid AN value 1 388348: May 9 17:43:31.502: HULC-MACsec: Invoking the callback for req BD0CCEC 388349: May 9 17:43:31.502: macsec_blocking_callback 388350: May 9 17:43:31.502: MACSEC_ERROR returned
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide