cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1615
Views
0
Helpful
2
Replies

MacSec instability on 3560CX

navlrac
Level 1
Level 1

I've been having stability issues with CTS manual MacSec on a pair of 3560-CX switches.

  • They are directly connected at 10G over ~4km of dark fiber.
  • With no MacSec, the link is perfect, no CRC issues, no dropped packets, etc.

Anyone encountered similar issues, or have suggestions?

 

With MacSec "cts manual", I have lots of issues including:

  1. A single link goes down randomly, for approx 205 seconds (once every 4-18 hours)
  2. Running two links LAG/LACP, link is lost more frequently (every 30min - 4hours), but port-channel recovers faster (30 sec)
  3. With LACP, when link goes down, I get LACP errors in the logs. With mode on, the link goes down for the same ~205 seconds.
  4. I see macsec replay fail / late packets (even though they are directly connected!! this shouldn't happen, right?):
                    rxL2UntaggedPkts = 57
                       rxL2NotagPkts = 0
                      rxL2SCMissPkts = 0
                        rxL2CTRLPkts = 0
                        rxL3CTRLPkts = 0
                   rxL3UnknownSAPkts = 0
                      rxL2BadTagPkts = 0
                    txL2UntaggedPkts = 0
                        txL2CtrlPkts = 0
                        txL3CtrlPkts = 0
                       txL3UnknownSA = 0

                            SA Index : 1
                  rxL2ReplayfailPkts = 210
                    rxL2AuthfailPkts = 0
                          rxL2PktsOK = 56586
                   rxL3AuthCheckFail = 0
                 rxL3ReplayCheckFail = 0
                      rxL2SAMissPkts = 0
                     rxL3EspGcm_Pkts = 0
                rxL3InverseCheckfail = 0
                       txL3Protected = 0
                       txL2Protected = 2172
#show macsec interface te1/0/1
 MACsec is enabled
  Replay protect : enabled
  Replay window : 0
  Include SCI : yes
  Use ES Enable : no
  Use SCB Enable : no
  Admin Pt2Pt MAC : forceTrue(1)
  Pt2Pt MAC Operational : no
  Cipher : GCM-AES-128
  Confidentiality Offset : 0

 Capabilities
  Identifier :
  Name :
  ICV length : 16
  Data length change supported: yes
  Max. Rx SA : 8
  Max. Tx SA : 8
  Max. Rx SC : 4
  Max. Tx SC : 4
  Validate Frames : strict
  PN threshold notification support : Yes
  Ciphers supported : GCM-AES-128

 Transmit Secure Channels
  SCI : ****************
  SC state : inUse(1)
   Elapsed time : 01:22:37
   Start time : 7w0d
   Current AN: 1
   Previous AN: 0
   Next PN: 912
   SA State: inUse(1)
   Confidentiality : no
   SAK Unchanged : no
   SA Create time : 4d15h
   SA Start time : 7w0d
   SC Statistics
    Auth-only Pkts : 0
    Auth-only Bytes : 0
    Encrypt Pkts : 84723
    Encrypt Bytes : 0
   SA Statistics
    Auth-only Pkts : 0
    Encrypt Pkts : 3083

  Port Statistics

 Receive Secure Channels
  SCI : ****************
  SC state : inUse(1)
   Elapsed time : 01:22:37
   Start time : 7w0d
   Current AN: 1
   Previous AN: 0
   Next PN: 971
   RX SA Count: 0
   SA State: inUse(1)
   SAK Unchanged : no
   SA Create time : 4d15h
   SA Start time : 7w0d
   SC Statistics
    Notvalid pkts 0
    Invalid pkts 0
    Valid pkts 848460
    Valid bytes 0
    Late pkts 210
    Uncheck pkts 0
    Delay pkts 14
    UnusedSA pkts 0
    NousingSA pkts 0
    Decrypt bytes 0
   SA Statistics
    Notvalid pkts 0
    Invalid pkts 0
    Valid pkts 57230
    UnusedSA pkts 0
    NousingSA pkts 0

  Port Statistics

Configuration on both switches:

interface TenGigabitEthernet1/0/1
 switchport trunk allowed vlan 5,7,20,21
 switchport trunk pruning vlan 2-4,6,8-19,22-1001
 switchport mode trunk
 cts manual
  no propagate sgt
  sap pmk ******************************** mode-list gcm-encrypt

 

#show cts interface te1/0/1
Global Dot1x feature is Disabled
Interface TenGigabitEthernet1/0/1:
    CTS is enabled, mode:    MANUAL
    IFC state:               OPEN
    Interface Active for 01:20:36.962
    Authentication Status:   NOT APPLICABLE
        Peer identity:       "unknown"
        Peer's advertised capabilities: "sap"
    Authorization Status:    NOT APPLICABLE
    SAP Status:              SUCCEEDED
        Version:             2
        Configured pairwise ciphers:
            gcm-encrypt

        Replay protection:      enabled
        Replay protection mode: STRICT

        Selected cipher:        gcm-encrypt

    Propagate SGT:           Disabled
    Cache Info:
        Expiration            : N/A
        Cache applied to link : NONE

    Critical-Authentication: Disabled
        Peer SGT: 0
        Peer SGT assignment: Untrusted
        Default PMK: Not Configured
        Default SGACL:
        Fail-Open: Enabled
    Statistics:
        authc success:              0
        authc reject:               0
        authc failure:              0
        authc no response:          0
        authc logoff:               0
        sap success:                1
        sap fail:                   0
        authz success:              0
        authz fail:                 0
        port auth fail:             0

    L3 IPM:   disabled.

 

2 Replies 2

navlrac
Level 1
Level 1

After a couple of dropped link events, the counters look as follows:

(See the replayfailed and rxL2AuthfailPkts / Notvalid pkts)

 

#show cts macsec counters interface te1/0/1
CTS Security Statistic Counters:
                    rxL2UntaggedPkts = 960
                       rxL2NotagPkts = 0
                      rxL2SCMissPkts = 0
                        rxL2CTRLPkts = 0
                        rxL3CTRLPkts = 0
                   rxL3UnknownSAPkts = 0
                      rxL2BadTagPkts = 0
                    txL2UntaggedPkts = 0
                        txL2CtrlPkts = 0
                        txL3CtrlPkts = 0
                       txL3UnknownSA = 0

                            SA Index : 1
                  rxL2ReplayfailPkts = 57460
                    rxL2AuthfailPkts = 777
                          rxL2PktsOK = 5618455
                   rxL3AuthCheckFail = 0
                 rxL3ReplayCheckFail = 0
                      rxL2SAMissPkts = 0
                     rxL3EspGcm_Pkts = 0
                rxL3InverseCheckfail = 0
                       txL3Protected = 0
                       txL2Protected = 4106
GENERIC Counters:
                      CRCAlignErrors = 0
                      UndersizedPkts = 0
                       OversizedPkts = 0
                        FragmentPkts = 0
                             Jabbers = 0
                          Collisions = 0
                            InErrors = 0
                           OutErrors = 0
                        ifInDiscards = 0
                   ifInUnknownProtos = 0
                       ifOutDiscards = 0
          dot1dDelayExceededDiscards = 0
                               txCRC = 0
                          linkChange = 0
#show macsec interface te1/0/1
 MACsec is enabled
  Replay protect : enabled
  Replay window : 0
  Include SCI : yes
  Use ES Enable : no
  Use SCB Enable : no
  Admin Pt2Pt MAC : forceTrue(1)
  Pt2Pt MAC Operational : no
  Cipher : GCM-AES-128
  Confidentiality Offset : 0

 Capabilities
  Identifier :
  Name :
  ICV length : 16
  Data length change supported: yes
  Max. Rx SA : 8
  Max. Tx SA : 8
  Max. Rx SC : 4
  Max. Tx SC : 4
  Validate Frames : strict
  PN threshold notification support : Yes
  Ciphers supported : GCM-AES-128

 Transmit Secure Channels
  SCI : ****************
  SC state : inUse(1)
   Elapsed time : 23:56:13
   Start time : 7w0d
   Current AN: 0
   Previous AN: 1
   Next PN: 1069
   SA State: inUse(1)
   Confidentiality : no
   SAK Unchanged : no
   SA Create time : 5d13h
   SA Start time : 7w0d
   SC Statistics
    Auth-only Pkts : 0
    Auth-only Bytes : 0
    Encrypt Pkts : 6772436
    Encrypt Bytes : 0
   SA Statistics
    Auth-only Pkts : 0
    Encrypt Pkts : 1068

  Port Statistics

 Receive Secure Channels
  SCI : ****************
  SC state : inUse(1)
   Elapsed time : 23:56:13
   Start time : 7w0d
   Current AN: 0
   Previous AN: 1
   Next PN: 1015
   RX SA Count: 0
   SA State: inUse(1)
   SAK Unchanged : no
   SA Create time : 5d13h
   SA Start time : 7w0d
   SC Statistics
    Notvalid pkts 92906
    Invalid pkts 0
    Valid pkts 1878968117
    Valid bytes 0
    Late pkts 57698
    Uncheck pkts 0
    Delay pkts 240
    UnusedSA pkts 0
    NousingSA pkts 0
    Decrypt bytes 0
   SA Statistics
    Notvalid pkts 0
    Invalid pkts 0
    Valid pkts 5257623
    UnusedSA pkts 0
    NousingSA pkts 0
#show interfaces te1/0/1
TenGigabitEthernet1/0/1 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is ********
  MTU 1550 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LR
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:04, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 21000 bits/sec, 8 packets/sec
  5 minute output rate 8000 bits/sec, 11 packets/sec
     342320448 packets input, 382844331049 bytes, 0 no buffer
     Received 1003179 broadcasts (986704 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 986704 multicast, 0 pause input
     0 input packets with dribble condition detected
     225839613 packets output, 166735018695 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

The issue still persists regularly :-/  TAC haven't been much help yet.

 

When the error occurs, I see the following suspicious bit in the logs:

 

388338: May  9 17:43:31.495: HULC-MACsec: Invoking the callback for req 1016AA8C
388339: May  9 17:43:31.495: macsec_blocking_callback
388340: May  9 17:43:31.495: Wake up the blocking process
388341: May  9 17:43:31.498: MACsec-CTS: Invoked for SA deletion
388342: May  9 17:43:31.498: MACsec API blocking the invoking context
388343: May  9 17:43:31.502: HULC-MACsec: Entering macsec_req_local_handler req BD0CCEC: 
388344: May  9 17:43:31.502: HULC-MACsec: macsec_process_req : Req type: 6
388345: May  9 17:43:31.502: HULC-MACsec: Process delete rxSA requestBD0CCEC for interface TenGigabitEthernet1/0/1
388346: May  9 17:43:31.502: HULC-MACsec: Deleting RxSA with AN = 1Curr AN = 0, Prev AN = 0
388347: May  9 17:43:31.502: Invalid AN value 1
388348: May  9 17:43:31.502: HULC-MACsec: Invoking the callback for req BD0CCEC
388349: May  9 17:43:31.502: macsec_blocking_callback
388350: May  9 17:43:31.502:  MACSEC_ERROR returned