Passing through on (Nexus 3k) VPC not working

[RichardS3]

Hi,

We have an issue where it looks like a VPC is not working very well with a port-channel on an ASR.
An ASR1001X connected to two Nexus switches, with a port-channel, is trying to reach a firewall which is connected to those switches.
The gateway of the ASR is one of the Nexus switches, which can reach the firewall again.
But as long as the interface on the ASR to the other Nexus (the one that is not the gateway) is up we cannot reach the firewall from the ASR.

Schematic our setup looks as following:

         /------ eth1/3 (po5) ---- Nexus 1 ---- eth1/6 (po4) ----\
  t0/0/0 (po1)                     |     |                        port34 (LAG0)
     |                  eth1/1 (po1)     eth1/2 (po1)                 |
     |                             |  V  |                            |
ASR1001X                           |  P  |                         Firewall
     |                             |  C  |                            |
     |                  eth1/1 (po1)     eth1/2 (po1)                 |
  t0/0/1 (po1)                     |     |                        port33 (LAG0)
        \------ eth1/3 (po5) ----- Nexus 2 ---- eth1/6 (po4) ----/

On the ASR we have the following (relevant) configuration:

interface Port-channel1
 no ip address
!
interface Port-channel1.147
 encapsulation dot1Q 147
 ip address 10.250.1.171 255.255.255.248
!
interface TenGigabitEthernet0/0/0
 description Nexus1-eth1/3
 no ip address
 cdp enable
 channel-group 1 mode active
!
interface TenGigabitEthernet0/0/1
 description Nexus2-eth1/3
 no ip address
 cdp enable
 channel-group 1 mode active
!

The gateway for the firewall on the ASR is 10.250.1.169 (which is an VLAN interface on Nexus2, which is connected to t0/0/1). The specific VLAN is configured to be allowed over the VPC peerlink on the Nexus switches.

When we try to ping the firewall we get a timeout, but when we shut port t0/0/0 it suddenly works. Therefor I assume VLAN 147 is not passed through from Nexus1 to Nexus2 over the VPC peerlink?

It seems we are doing something fundamentally wrong but we cannot figure out what it is.

This is the (relevant) configuration on Nexus1:

vpc domain 2
  peer-keepalive destination 10.251.101.14

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  spanning-tree port type network
  vpc peer-link

interface port-channel5
  description ASR1001
  switchport mode trunk
  switchport trunk allowed vlan 102,109,145,147,1100-1101,1103
  vpc 5

interface Ethernet1/1
  description VPC
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  channel-group 1 mode active

interface Ethernet1/2
  description VPC
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  channel-group 1 mode active

interface Ethernet1/3
  description ASR1001-T0/0/0
  switchport mode trunk
  switchport trunk allowed vlan 102,109,145,147,1100-1101,1103
  channel-group 5 mode active

And this is the (relevant) configuration on Nexus2, which is almost identical except for an extra VLAN interface:

vpc domain 2
  peer-keepalive destination 10.251.101.13

interface Vlan147
  no shutdown
  mtu 9000
  ip address 10.250.1.169/29

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  spanning-tree port type network
  vpc peer-link

interface port-channel5
  description ASR1001
  switchport mode trunk
  switchport trunk allowed vlan 102,109,145,147,1100-1101,1103
  vpc 5

interface Ethernet1/1
  description VPC
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  channel-group 1 mode active

interface Ethernet1/2
  description VPC
  switchport mode trunk
  switchport trunk allowed vlan 102-103,107,109,145,147,1100-1101,1103
  channel-group 1 mode active

interface Ethernet1/3
  description ASR1001-T0/0/1
  switchport mode trunk
  switchport trunk allowed vlan 102,109,145,147,1100-1101,1103
  channel-group 5 mode active

On both port-channels VLAN 147 is allowed so it should be reachable to reach Nexus2 over Nexus1 from ASR1001?

The VPC seems working fine as well:

Nexus1# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 2   
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : failed  
Type-2 inconsistency reason       : SVI type-2 configuration incompatible
vPC role                          : primary, operational secondary
Number of vPCs configured         : 4   
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans    
--    ----   ------ -------------------------------------------------
1     Po1    up     102-103,107,109,145,147,1100-1101,1103                               

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
5     Po5           up     success     success               102,109,145,14              
                                                             7,1100-1101,11              
                                                             03                          
           

Can anyone point out what we are doing wrong here?

Thanks in advance !

/ Richard

[Reza Sharifi]

Hi,

What is the reason for not having a vlan interface (SVI) on Nexus-1?

interface Vlan147
  no shutdown
  mtu 9000
  ip address 10.250.1.x/29

I also see this in the output of sh vpc:

Type-2 consistency status         : failed  
Type-2 inconsistency reason       : SVI type-2 configuration incompatible

HTH

[RichardS3]

Hi Reza,

Thank you for your answer.

We used to have a SVI, in the same VLAN, on Nexus1 as well but while reading this forum we read somewhere it would be better to have two seperate VLANs.

On Nexus1 we have VLAN145 now and on Nexus2 we have VLAN147. There is a BGP session between the ASR and both switches over those VLANs. As long as Nexus1 is the preferred path everything works fine. If the BGP session with Nexus1 goes down the preferred path will become Nexus2. But if, for whatever reason, the link stays up with Nexus1 we have the problem illustrated in my first post.

When we had both switches in the same VLAN we had the exact same problem.

The warning about the SVI type-2 configuration incompatible is because of we don't have the same SVI's on both switches. Which is as far as I read only a warning and not an error.

Nexus1# sh vpc consistency-parameters global 

    Legend:
        Type 1 : vPC will be suspended in case of mismatch

Name                        Type  Local Value            Peer Value             
-------------               ----  ---------------------- -----------------------
STP MST Simulate PVST       1     Enabled                Enabled               
STP Port Type, Edge         1     Normal, Disabled,      Normal, Disabled,     
BPDUFilter, Edge BPDUGuard        Disabled               Disabled              
STP MST Region Name         1     ""                     ""                    
STP Disabled                1     VLANs 1-3967           VLANs 1-3967          
STP Mode                    1     Rapid-PVST             Rapid-PVST            
STP Bridge Assurance        1     Enabled                Enabled               
STP Loopguard               1     Disabled               Disabled              
STP MST Region Instance to  1                                                  
 VLAN Mapping                                                                  
STP MST Region Revision     1     0                      0                     
Interface-vlan admin up     2     102-103,107,145        103,107,109,147       
Interface-vlan routing      2     102-103,107,145        103,107,109,147       
capability                                                                     
Xconnect Vlans              1                                                  
QoS (Cos)                   2     ([0-7], [], [], [],    ([0-7], [], [], [],   
                                  [], [])                [], [])               
Network QoS (MTU)           2     (9000, 9000, 9000,     (9000, 9000, 9000,    
                                  9000, 9000, 9000)      9000, 9000, 9000)     
Network Qos (Pause:         2     (F, F, F, F, F, F)     (F, F, F, F, F, F)    
T->Enabled, F->Disabled)                                                       
Input Queuing (Bandwidth)   2     (100, 0, 0, 0, 0, 0)   (100, 0, 0, 0, 0, 0)  
Input Queuing (Absolute     2     (F, F, F, F, F, F)     (F, F, F, F, F, F)    
Priority: T->Enabled,                                                          
F->Disabled)                                                                   
Output Queuing (Bandwidth   2     (100, 0, 0, 0, 0, 0)   (100, 0, 0, 0, 0, 0)  
Remaining)                                                                     
Output Queuing (Absolute    2     (F, F, F, F, F, F)     (F, F, F, F, F, F)    
Priority: T->Enabled,                                                          
F->Disabled)                                                                   
HW profile Forwarding Mode  1     normal                 normal                
Allowed VLANs               -     102-103,107,109,145,14 102-103,107,109,145,14
                                  7,1100-1101,1103       7,1100-1101,1103      
Local suspended VLANs       -     -                      -               

The vlan 107 and 109 difference is correct as certain devices are only connected to nexus1 (vlan107) and other devices are only connected to nexus2 (vlan109).

[balaji.bandi]

If you configure only on one nexus - that becomes an orphan port, vPC need to be active/active, why not considering HSRP config here for  vlan147 

interface Vlan147
  no shutdown
  mtu 9000
  ip address 10.250.1.169/29

[RichardS3]

Hello Balaji,

The reason there is no HSRP (/VRRP) is because the interface is used for BGP peering. If we use HSRP the IP-address will be moved to the other switch, in case of a failover, but the BGP session would become invalid and needs to be re-initialized again.

We used to have the same SVI on both switches but it came with the exact same problem.

[balaji.bandi · Accepted solution]

If you Loking some Layer 3, you can not use peerlink, you need consider having Layer 3 as per vPC design concern for the BGP (is this iBGP or eBGP)

[RichardS3]

It is eBGP and we have indeed reconsidered our design .

We have removed the port-channel on the ASR and on the Nexus switches.

Instead we are using individual VLANs to each switch and not forwarding this VLAN over the VPC:

        /--- eth1/3 - vlan145 ---- Nexus 1 ---- eth1/6 (po4) ----\
  t0/0/0.145                       |     |                        port34 (LAG0)
     |                  eth1/1 (po1)     eth1/2 (po1)                 |
     |                             |  V  |                            |
ASR1001X                           |  P  |                         Firewall
     |                             |  C  |                            |
     |                  eth1/1 (po1)     eth1/2 (po1)                 |
  t0/0/1.147                       |     |                        port33 (LAG0)
        \--- eth1/3 - vlan147 ---- Nexus 2 ---- eth1/6 (po4) ----/

BGP will automatically change the route if one of the Nexus goes down.

[balaji.bandi]

yes, that is also a good approach BGP and BFD combination also if you prefer to do.

[MHM Cisco World]

...

[MHM Cisco World]

Hi friend
please what is model of nexus you use it is 3... ?