vPC Etherchannel between Nexus and Arista switches(LACP)

praun · ‎10-13-2015

Would you recommend this setup instead.

Alt .1

On Nexus:

Create the vPC 1830(Arista #1) and corresponding Port-channel + interface 8/30 on both Nexus)

Create the vPC 1831(Arista #2) and corresponding Port-channel + interface 8/31 on both Nexus)

and use an ordinary ether-channel uplink on each Arista. And quit the MLAG between the 2 Arista.

Would this be a better solution?

Trying to create Etherchannel to Arista switches.

vPC between nexus are up, MLAG between Aristas are up.

From Nexus, one side is up and connected. After port-chanel shut/no shut. both sides from nexus is in state linkFlapE.

See Nexus and arista configs below.

Has anyone seen this problem? What is misconfigured?

Nexus config (both sides equal)
!
interface port-channel1830
switchport
switchport access vlan 3842
spanning-tree port type normal
spanning-tree bpduguard enable
mtu 9216
storm-control broadcast level 10.00
storm-control multicast level 10.00
vpc 1830
!
interface Ethernet8/30
switchport access vlan 3842
spanning-tree port type normal
mtu 9216
storm-control broadcast level 10.00
storm-control multicast level 10.00
channel-group 1830 mode active
no shutdown
!
!
!
Arista config (one side)
vlan 4000
   trunk group mlagpeer
!
interface Port-Channel20
   description mlag peer connection
   switchport mode trunk
   switchport trunk group mlagpeer
!
interface Port-Channel100
   switchport mode access
   mlag 100
!
interface Ethernet1
   description uplink1
   channel-group 100 mode active
!
interface Ethernet2
   description uplink2
   channel-group 100 mode active
!
interface Vlan4000
   description mlag peer link interface
   no autostate
   ip address 10.0.0.2/30
!
mlag configuration
   domain-id nan-mlag
   local-interface Vlan4000
   peer-address 10.0.0.1
   peer-link Port-Channel20
!

Mark Malone · ‎10-13-2015

I would have thought the Nexus E8/30 interfaces going to Arista would be terminating on 1 switch not 2 locally and the Nexus side configured as vpc going back to the arista if your trying to create etherchannel as you cant really have them split across multiple switches in that vpc design , I could be wrong but that's the ways ours are setup and they work with the likes of dell,hps etc on the connecting side to the nexus switches

praun · ‎10-13-2015

Hi Mark.

I have made an alternative as i understand you. Then i don't have to use multi chassis Arista (MLAG)

See new picture.

Is this setup working for you?

Regards

Steve Fuller · ‎10-13-2015

Hi,

In a similar vein to Mark, I don’t quite understand the connectivity you have.

If you have Ethernet 8/30 from both N7010 left and N7010 right connected to a single Arista switch, then whilst it’s vPC on the Nexus, it’s not MLAG on the Arista. This should simply be a standard LAG on the Arista.

You would only need MLAG on the Arista if you’re connecting as shown in the diagram with N7010 left Ethernet 8/30 connected to Arista left Ethernet 1, and N7010 right Ethernet 8/30 connected to Arista right Ethernet 1. Then you would need to have Ethernet 1 interface of both Arista switches configured as part of the MLAG.

Regards

praun · ‎10-13-2015

Hi Steve.

I see your point. The only reason Eth 1 and Eth2 on each arista are parts of Channel-group 100 is that they have prepared 2x10G to each Nexus.

As shown in the picture there are 2 Arista switches connected to 2 nexus 7010 switches.

Eth 1 from arista 1 goes to nexus 1

Eth 1 from arista 2 goes to nexus 2

Eth 21-24 interconnects the 2 Arista. in MLAG

Se config below.

Thanks for your interrest.

Arista config.

interface Ethernet21

description mlag peer connection

channel-group 20 mode active

!

interface Ethernet22

description mlag peer connection

channel-group 20 mode active

!

interface Ethernet23

description mlag peer connection

channel-group 20 mode active

!

interface Ethernet24

description mlag peer connection

channel-group 20 mode active

!

interface Vlan4000

description mlag peer link interface

no autostate

ip address 10.0.0.2/30

!

no ip routing

!

mlag configuration

domain-id nan-mlag

local-interface Vlan4000

peer-address 10.0.0.1

peer-link Port-Channel20

!

Steve Fuller · ‎10-13-2015

Hi,

I’m still not clear on how the configurations match the diagram. The diagram seems to indicate port-channel 1830 as the vPC peer-link i.e., it’s shown on the link between the Nexus 7010, but then you’re showing Ethernet 8/30 with channel-group 1830 i.e., part of port-channel 1830 using vpc 1830 towards the Arista.

Can you post a show vpc from both Nexus switches?

If it helps, I previously posted a working configuration to the post Nexus vPC to Arista MLAG so perhaps you can use that as a template?

Regards

praun · ‎10-13-2015

Hi again.

Thansk for your template

Here is the show vPC 1830 and show interface status from both Nexus
I have also enclosed a chart, comparring the 2 Arista and the Nexus as well.
I see no differences from your working example.
Port E1 from each Arista is linked to E8/30 on each Nexus

Nexus #1
sh vpc 1830
vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
1830 Po1830 up     success     success                    3842

sh inter e8/30 status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth8/30 EMC-DC1-M30 connected 3842 full 10G 10Gbase-LR

Nexus #2
sh vpc 1830
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
1830 Po1830 down* success success -----

sh int e8/30 status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth8/30 EMC-DC1-M30 linkFlapE 3842 auto auto 10Gbase-LR

One port on Nexus always retur to status linkFlapE

Arista 1	Arista 2
!
vlan 4000	vlan 4000
trunk group mlagpeer	trunk group mlagpeer
!	!
interface Vlan4000	interface Vlan4000
description mlag peer link interface	description mlag peer link interface
no autostate	no autostate
ip address 10.0.0.2/30	ip address 10.0.0.1/30
!	!
mlag configuration	mlag configuration
domain-id nan-mlag	domain-id nan-mlag
local-interface Vlan4000	local-interface Vlan4000
peer-address 10.0.0.1	peer-address 10.0.0.2
peer-link Port-Channel20	peer-link Port-Channel20
!	!
interface Port-Channel20	interface Port-Channel20
description mlag peer connection	description mlag peer connection
switchport mode trunk	switchport mode trunk
switchport trunk group mlagpeer	switchport trunk group mlagpeer
!
interface Ethernet21	interface Ethernet21
description mlag peer connection	description mlag peer connection
channel-group 20 mode active	channel-group 20 mode active
!	!
interface Ethernet22	interface Ethernet22
description mlag peer connection	description mlag peer connection
channel-group 20 mode active	channel-group 20 mode active
!	!
interface Ethernet23	interface Ethernet23
description mlag peer connection	description mlag peer connection
channel-group 20 mode active	channel-group 20 mode active
!	!
interface Ethernet24	interface Ethernet24
description mlag peer connection	description mlag peer connection
channel-group 20 mode active	channel-group 20 mode active
!	!
interface Port-Channel100	interface Port-Channel100
description Uplink	description Uplink
switchport mode access	switchport mode access
mlag 100	mlag 100
!	!
interface Ethernet1	interface Ethernet1
description Uplink1	description Uplink1
channel-group 100 mode active	channel-group 100 mode active
!	!
interface Ethernet2	interface Ethernet2
description Uplink2	description Uplink2
channel-group 100 mode active	channel-group 100 mode active
!	!

Nexus 7010 #1	Nexus 7010 #2
!	!
vpc domain 5
peer-switch	vpc domain 5
role priority 1000	role priority 2000
system-priority 3	system-priority 3
peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf vPC-peer-keepalive	peer-keepalive destination 10.1.1.1 source 10.1.1.2 vrf vPC-peer-keepalive
peer-gateway	peer-gateway
auto-recovery	auto-recovery
ip arp synchronize	ip arp synchronize

interface port-channel10	interface port-channel10
description vPC peer-link	description vPC peer-link
switchport	switchport
switchport mode trunk	switchport mode trunk
switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899	switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899
spanning-tree port type network	spanning-tree port type network
mtu 9216	mtu 9216
port-channel port load-defer	port-channel port load-defer
vpc peer-link	vpc peer-link
!	!
interface Ethernet1/26	interface Ethernet1/26
switchport	switchport
switchport mode trunk	switchport mode trunk
switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899	switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899
rate-mode dedicated force	rate-mode dedicated force
mtu 9216	mtu 9216
channel-group 10 mode active	channel-group 10 mode active
no shutdown	no shutdown
!	!
interface Ethernet2/26	interface Ethernet2/26
switchport	switchport
switchport mode trunk	switchport mode trunk
switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899	switchport trunk allowed vlan 1101-1400, 2000-2399,2500-3399,3500-3899
rate-mode dedicated force	rate-mode dedicated force
mtu 9216	mtu 9216
channel-group 10 mode active	channel-group 10 mode active
no shutdown	no shutdown
!	!
interface port-channel1830	interface port-channel1830
inherit port-profile xxxx	inherit port-profile xxxx
description EMC-DC1-M30	description EMC-DC1-M30
switchport access vlan 3842	switchport access vlan 3842
spanning-tree bpduguard disable	spanning-tree bpduguard disable
vpc 1830	vpc 1830
!	!
interface Ethernet8/30	interface Ethernet8/30
description EMC-DC1-M30	description EMC-DC1-M30
switchport access vlan 3842	switchport access vlan 3842
mtu 9216	mtu 9216
storm-control broadcast level 10.00	storm-control broadcast level 10.00
storm-control multicast level 10.00	storm-control multicast level 10.00
channel-group 1830 mode active	channel-group 1830 mode active
no shutdown	no shutdown

Steve Fuller · ‎10-13-2015

In the original post you said "After port-chanel shut/no shut. both sides from nexus is in state linkFlapE". I took this to mean the interface on both Nexus were becoming disabled.

Is it only the one interface on Nexus #2 that becomes disabled? If so this could simply be a media issue, either SFP+ transceiver or fibre. Is it possible to get those checked?

Regards

praun · ‎10-13-2015

Hi Steve.

I don't think it's a fiber issue. We have 3 sites with the same problems.

It's likly a config problem.

We have tested the fiber and changed SFP's without a change.

The link is behaving strange. Here is a print from the log.
As i Said. The fiber is testet and the SFP+ has been changed:
It toggles between active and link failure several times until error disabled.

Se log

%ETHPORT-5-IF_UP: Interface Ethernet8/30 is up in mode access
%ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet8/30 is down (Link failure)
%ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet8/30 is down
%ETHPORT-5-SPEED: Interface Ethernet8/30, operational speed changed to 10 Gbps
%ETHPORT-5-IF_DUPLEX: Interface Ethernet8/30, operational duplex mode changed to Full
%ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet8/30, operational Receive Flow Control state changed to off
%ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet8/30, operational Transmit Flow Control state changed to off
%ETHPORT-5-SPEED: Interface port-channel1830, operational speed changed to 10 Gbps
%ETHPORT-5-IF_DUPLEX: Interface port-channel1830, operational duplex mode changed to Full
%ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel1830, operational Receive Flow Control state changed to off
%ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel1830, operational Transmit Flow Control state changed to off
%ETH_PORT_CHANNEL-4-PORT_INDIVIDUAL: port Ethernet8/30 is operationally individual
%ETHPORT-5-IF_UP: Interface Ethernet8/30 is up in mode access
%ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet8/30 is down (Link failure)
%ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet8/30 is down
%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet8/30 is down (Error disabled. Reason:Too many link flaps)

Steve Fuller · ‎10-13-2015

Agreed. It'd be pretty unlucky if this was a fibre / transceiver issue :-)

The error %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN could be indicating that the Nexus is not receiving LACPDU (Link Aggregation Control Protocol Data Units) and so moves the port into what the IEEE standard calls an Individual (I) state. That said I've not seen interfaces disabled due to link flap as a result of that.

What do you see on each switch, both Nexus and Arista, if you run the show lacp counters command? Can you see the counters increasing in all cases or only on those ports that are not being disabled.

Can you also add the command logging timestamps [seconds|milliseconds] to the configuration so we have an idea of time frame between link state changes?

Regards

praun · ‎10-23-2015

Hi Steve.

Thanks for your reply. Here is an update.
We can now get the link connected by doing a interface shut on both sides. Then first open the Nexus interface and then the Arista interface before the Nexus interface goes into LinkFlap-errordisabled - and the link is up/connected without problems. This is the only way to get connection. So it's likly a LACP problem.

jnrs-sev-109-0201# sh lacp counters int po 1830
                    LACPDUs         Marker      Marker Response    LACPDUs
Port              Sent   Recv     Sent   Recv     Sent   Recv      Pkts Err
---------------------------------------------------------------------
port-channel1830
Ethernet8/30       124637 34251    0      0        0      0        0
We have lots of LACP count stepping up.

Shut/noshut on Nexus while Ariste interface is enabled won't bring the interface up.
EMC who is responsible for the Arista switches has verified the Nexus and Arista configs and can't find any misconfigurations. So the problem has been escalatedto Arista tech-support.

Steve Fuller · ‎10-24-2015

Hi,

Thanks for posting back. I'd be interested to hear what the vendor has to say as I've never had any issues with Cisco vPC to Arista MLAG.

One difference I noticed when I went through this again was that you've configured the port-channel with switchport mode access, but whenever I've configured this I've always used switchport mode trunk.

Regards

peter.skovgaard.svane · ‎06-10-2016

Did you ever get a resolution to this problem? I have an almost identical setup (with EMC ECS connected through EMC provided Arista switches, MLAGed to a Nexus 7010 VPC pair) and am experiencing some similar problems.

Steve Fuller · ‎10-13-2015

Hi,

You said:

"On Nexus:

Create the vPC 1830(Arista #1) and corresponding Port-channel + interface 8/30 on both Nexus)

Create the vPC 1831(Arista #2) and corresponding Port-channel + interface 8/31 on both Nexus)

and use an ordinary ether-channel uplink on each Arista. And quit the MLAG between the 2 Arista.

Would this be a better solution?"

I would continue with the vPC to MLAG configuration. It works well in my experience and means you get the benefits associated with MLAG i.e., no STP and all links forwarding.

I'd go for a "double sided" MLAG where all four link between the Nexus pair and the Arista pair are part of the same Cisco vPC / Arista MLAG port-channel group.

Based on what you've provided I think the following is the required configuration:

Nexus:

Nexus 7K1

Nexus 7K2

#
feature vpc
#
vpc domain <domain_id>
  peer-switch
  role priority 1
  peer-keepalive destination <peer_ip_address>
  delay restore 60
  peer-gateway
#
interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
#
interface Ethernet <mod>/<port>
  switchport mode trunk
  channel-group 1 mode active
#
interface Ethernet <mod>/<port>
  switchport mode trunk
  channel-group 1 mode active
#
interface port-channel1830
  switchport mode access
  vpc 1830
#
interface Ethernet8/30
  switchport mode access
  channel-group 1830 mode active
#
interface Ethernet8/31
  switchport mode access
  channel-group 1830 mode active
#

#
feature vpc
#
vpc domain <domain_id>
  peer-switch
  role priority 1
  peer-keepalive destination <peer_ip_address>
  delay restore 60
  peer-gateway
#
interface port-channel1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
#
interface Ethernet <mod>/<port>
  switchport mode trunk
  channel-group 1 mode active
#
interface Ethernet <mod>/<port>
  switchport mode trunk
  channel-group 1 mode active
#
interface port-channel1830
  switchport mode access
  vpc 1830
#
interface Ethernet8/30
  switchport mode access
  channel-group 1830 mode active
#
interface Ethernet8/31
  switchport mode access
  channel-group 1830 mode active
#

Arista:

Arista #1

Arista #2

!
vlan 4000
   trunk group mlagpeer
!
interface Vlan4000
   ip address ip address 10.0.0.1/30
!
mlag configuration
   domain-id nan-mlag
   local-interface Vlan4000
   peer-address 10.0.0.2
   peer-link Port-Channel20
!
interface Port-Channel20
   switchport mode trunk
   switchport trunk group mlagpeer
!
interface Ethernet <port_x>
   channel-group 20 mode active
!
interface Ethernet <port_y>
   channel-group 20 mode active
!
interface Port-Channel100
   switchport mode access
   mlag 100
!
interface Ethernet1
  switchport mode access
  channel-group 100 mode active
!
interface Ethernet2
  switchport mode access
  channel-group 100 mode active
!

!
vlan 4000
   trunk group mlagpeer
!
interface Vlan4000
   ip address ip address 10.0.0.2/30
!
mlag configuration
   domain-id nan-mlag
   local-interface Vlan4000
   peer-address 10.0.0.1
   peer-link Port-Channel20
!
interface Port-Channel20
   switchport mode trunk
   switchport trunk group mlagpeer
!
interface Ethernet <port_x>
   channel-group 20 mode active
!
interface Ethernet <port_y>
   channel-group 20 mode active
!
interface Port-Channel100
   switchport mode access
   mlag 100
!
interface Ethernet1
  switchport mode access
  channel-group 100 mode active
!
interface Ethernet2
  switchport mode access
  channel-group 100 mode active
!

As an aside, I'd probably be tempted to operate the links between the Nexus and Arista as IEEE 802.1Q trunks. If you only want the single VLAN 3842 at this stage simply add the command switchport trunk allowed vlan 3842 to your port-channel configuration. At least this way if you want to have additional VLANs at a later stage you can simply add them in a way that won't cause any disruption.

Regards

praun · ‎10-13-2015

Thats correct. I see your point.

But configuring the ports as access ports should work as well.

Agree?

/Regards