MACSec across QinQ provider network

[n-russell-biggie]

Hi,

I have three switches (3650's), in three different locations connected together through service provider circuits.

The provider gives me QinQ tunnels in order for me to trunk my switches together. They also tell me  that the tunnels are fully transparent and effectively means I have point-to-point connections for my switches.

Prior to MACSec configuration CDP neighbors show our switches as neighbors and I see no provider switch info.

On that basis of this I have deployed manual MACSec on the trunk interfaces of my switches with the below commands:

#cts manual

##no propergate sgt

## sap pmk [KEY] mode-list gcm-encrypt

We are unable to get the trunks to come up. We see the line protocol as "down"

In tests with the switches physically connected together in a lab the manual MACSec works perfectly.

Is it the case that running MACSec across a providers network in a QinQ tunnel is not possible or is there a workaround?

Thanks

Nick

[Bogdan Nita]

MACSec over QinQ will not work, MACsec is desigened to protect communication between directly connected trusted components, so it will not work over multiple hops.

[bohumir.hejduk]

Hi Bogdan Nita, are you pretty sure that MACSec on QinQ leased line will not work? Did you test that? I had a chat with Cisco representative and he said that he doesn't see any problem with running MACSec on Q-in-Q ISP WAN line. Bohumir

[n-russell-biggie]

Hello,

Actually we did get this to work across a providers network that I believe is using QinQ. We had three switches connected together between an office and two data centers. The MACSec all came up correctly.

The issue we have now is that when we reboot one of the switches in the triangle it appears that for 30 minutes we get uni-directional issues and broadcast storms. This lasts approx 30 minutes and then the MACSec appears to work again and the network stabilizes.

We have a TAC case open for this and will be carrying out some troubleshooting on Tuesday evening. I will post any relevant updates to here following the troubleshooting with TAC.

Thanks

Nick

[Bogdan Nita]

Hi Nick,

Thanks for sharing the information.
Would be interesting to know how you configured MACSec and how is your service provider forwarding the packets.

As far as I understand MACSec as described in 802.1AE–2006 does not offer the possibility (at least in theory) to run MACsec over PBNs.
Here is a paper on it: http://www.ieee802.org/1/files/public/docs2013/ae-seaman-macsec-hops-0626-v03.pdf

That being said it, Cisco has added additional capabilities to MACSec in order to be able to run MACSec over PBNs:
The WAN MACsec offering is standards based but offers additional capabilities not found in earlier MACsec capabilities. More specifically, MACsec can be leveraged by enterprise customers over public carrier Ethernet offerings, allowing customers to adapt to the public carrier Ethernet service offering and capabilities (or restrictions).
New enhancements for WAN MACsec include:
1. 802.1Q Tag in the Clear
2. Standard IEEE 802.1X-rev MACsec Key Agreement
3. Integrated MACsec authentication adaptability over public Carrier Ethernet transport
https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/identity-based-networking-services/white-paper-c11-737544.html

Regards,
Bogdan

[n-russell-biggie]

Hi Bogdan,

The three switches are connected in a triangle. One switch in each of the two Data Centres and one switch in the clients office.

MACsec is configures as below on the trunks to each of the switches:

interface GigabitEthernet1/1/1

description Trunk xxxxxxxx Gig 1/1/1

switchport mode trunk

cts manual

no propagate sgt

sap pmk A0646E7153F9837DDB992AA4983852255181B17B71D0A2D55AFBEBE267F8D76D mode-list gcm-encrypt

spanning-tree guard loop

end

I am unsure on what the providers are using to connect the three switches together but I would gues dot1q or the likes. We did supply them with ethertype codes that are detailed in MACsec documentation but am not sure which ones if any they are using.

MACsec appears to work perfectly in this scenario. However, the client states that when a reload of one of the switches is carried out they experience uni-directional traffic and broadcast storms. We are working with TAC to try and replicate the issues but so far we are unable to do so and MACsec is functioning correctly.

Thanks

Nick