NEXUS - Inter-site connectivity problem

[peters97]

Hello All,

We have a 2 sites in different datacenters, let's say A1 site and A2 site.

A1 site has 2x Nexus 9k in vPC, A2 site has 1x Nexus 9k.

Sites have two inter-connect links between them, running LACP (1x port from each A1 nexus to 2x port on one A2 nexus)

We have multiple VRF with networks for servers. Each VRF has its own VLAN used as Inter-Connect between A1 and A2.

(server->A1nexus->Inter-Connect Network->A2nexus->server)

There's a problem when we do upgrade or reboot of A1 nexus switches.

Several servers from A1 site has a problem to e.g. ping A2 server (or connect to some service). Before reboot/upgrade it was working.

However, when we run traceroute from A1 server and then try ping it starts working. 

Traceroute solves the problem (but it's not ideal solution).

Do you have any idea what could be the problem? (ARP) problem in Inter-Connect networks between A1 and A2?

Thank you.

[MHM Cisco World]

Server in both site in same VLAN or different?

How you interconnect the two site via vpc double sided ?

MHM

[peters97]

Different VLAN.

Example: A1 side - server in vlan 125, Interconnect vlan 500, A2 side - server in vlan 225

I'm sorry, I don't understand second question. A1 nexus switches are in vPC. Sending picture if it helps.

[MHM Cisco World]

Ok let start'

2xN9K is run vpc 

And connect to 1N9K via PO config as vpc ?

In Site 1N9K the gw of host is 1N9K?

In site 2xN9K the gw of host is VIP of hsrp?

 You use IGP or static route between two site?

MHM

[peters97]

And connect to 1N9K via PO config as vpc ?  -> yes, Port-Channel has vpc configured

In Site 1N9K the gw of host is 1N9K? -> Yes

In site 2xN9K the gw of host is VIP of hsrp? -> VIP of HSRP

You use IGP or static route between two site? -> Static routes, in IC network, for 1N9K next hop to A1 site is VIP of HSRP 

[MHM Cisco World]

All above is OK'  And correct still two points must check

You mention that the stp is mess'

The stp bridge assurance' check if it disable port 

Use peer-gateway in 2xN9K vpc domain' it can the role is change after upgrading.

MHM

[balaji.bandi]

Example: A1 side - server in vlan 125, Interconnect vlan 500, A2 side - server in vlan 225  - not sure where this gateway resides.

best is move to HSRP v2 and make sure Peer-gateway configured

PoX going to Site A2 in vPC that should work as expected.

we need also see your confiuration and stp / vpc config, and HSRP config and output if possible to share.

[balaji.bandi]

How is your Layer 3 Interface configured on vPC devices ? HSRP ?

what kind of spanning tree are you using  "network" ?

A2 site connected to A1 side the link are configured that port-channel as vPC ?

[peters97]

Yes, L3 vlan interfaces, HSRP version 1, one of the A1 nexuses has priority 110, second default priority, preempt enabled, other hsrp options in default

Unfortunately, spanning tree on A1 side nexuses is mess, towards to A2 side -> port-channel are type normal, but physical interfaces in that port-channel are type network.

Yes, A1 side nexuses have port-channel in vPC.