OTV adjacency failure

[Ulrich Hansen]

Hello,

I'm in the process of deploying N7K in my two datacenters and have run into a problem with OTV.

The topology is quite simple:

- 4 Cat6509 as Core-routers connecting the two sites. The Core network operates as a full-mesh Layer3-core, with OSPF as IGP and mpls enabled (though not used yet).

- Each datacenter has a pair of N7K's L2-interconnected and with vPC running.

- On each N7K, a dedicated vdc has been created for OTV and each vdc connects to the "main" vdc through a vPC channel

All OTV vdc's have been created using the same configuration (with the obvious exceptions of ip-adresses) for consistency. Each vdc has been configured with two overlay-interfaces with the extendable vlans split 50/50 among them.

The two OTV-vdc's in Site1 establish ajcacencies without any problems. Both toward Site2 and locally across the site-vlan. AED is negotiated without any problems.

But at Site2, only one OTV-vdc form adjacency with Site1. When the fourth vdc is created and configured, it briefly forms adjacency with only one of the OTV-edges at Site1 and as soon as the local OTV-edge neighborship is established, the external adjacencies are torn down and subsequent attempts to re-establish them, result in a state INIT.

A display of the igmp groups on the core-router shows, that the failing OTV-edge device has joined the control-group, but still the adjacency cannot be re-established. The otv-isis internal log shows no sign of mismatch of any kinds or auth.failures.

I've tried to remove the entire OTV/vdc config on the failing device, reload and re-apply it, but the same chain of events takes place. I've gone through each N7K, comparing configurations, making sure that site-vlans are not blocked by spanning-tree and are allowed on the individual interfaces. All checks, so I'm stuck at a somewhat dead end.

I'm running NXOS v5.1 on all devices.

Any thoughts or suggestions ?

Thanks

/Ulrich

[Amit Singh]

Ulrich,

Why are you creating two overlays for this configuration? Please have all these OTV VDC's or both the sites within the same Overlay. Also, make sure that you are using the same multicast Control and data groups.

Please could you paste the toplogy diagram and configurations, if the same overlay configuration doesnot work.

HTH,

-amit singh

[Ulrich Hansen]

Hi Amit,

Thanks for replying.

My reasons for using two distinct Overlay's with different mcast-addresses is based on a the assumption, that this would create a more evenly distributed load across the core-network (I could be wrong though). Even if the AED negotiation spreads the extended vlans fairly mutual between the two edge-devices, using the same data-group address, won't this cause the traffic to always follow the same physical path through the core-network?

However, I have to admit, that this was caused by a simple typo (duplicate ip's on two edge-devices, my bad). I've corrected the error, and I'm now seeing all adj. formed succesfully, even with multible overlays.

The only thing that still have me wondering is, that two of the sites have both local and external adjacencies, as listed below (this is taken from the edge-device that was failing before):

DC2SD02-OTV# sh otv adj
Overlay Adjacency database

Overlay-Interface Overlay1  :
Hostname                              System-ID      Dest Addr       Up Time   State
DC2SD01-OTV                      b414.89e0.ea42 10.1.1.217      00:08:33  UP
DC1SD02-OTV                      b414.89dc.2ec2 10.1.1.225      00:08:33  UP
DC1SD01-OTV                      b414.89dd.c8c2 10.1.1.229      00:08:33  UP

Overlay-Interface Overlay2  :
Hostname                              System-ID      Dest Addr       Up Time   State
DC2SD01-OTV                      b414.89e0.ea42 10.1.1.217      00:08:32  UP
DC1SD02-OTV                      b414.89dc.2ec2 10.1.1.225      00:08:32  UP
DC1SD01-OTV                      b414.89dd.c8c2 10.1.1.229      00:08:32  UP

DC2SD02-OTV# sh otv site

Site Adjacency Information (Site-VLAN: 3000) (* - this device)

Overlay1 Site-Local Adjacencies (Count: 2)

  Hostname                         System-ID           Up Time   Ordinal
  -------------------------------- -------------- --------- ----------
* DC2SD02-OTV                      108c.cf18.1642 00:09:53  0
  DC2SD01-OTV                      b414.89e0.ea42 00:12:45  1

Overlay2 Site-Local Adjacencies (Count: 2)

  Hostname                         System-ID           Up Time   Ordinal
  -------------------------------- -------------- --------- ----------
* DC2SD02-OTV                      108c.cf18.1642 00:09:51  0
  DC2SD01-OTV                      b414.89e0.ea42 00:12:43  1

The two external adj with DC2SD01-OTV should not be there, as they are local neighbors.

/Ulrich

Message was edited by: Ulrich Hansen

[Amit Singh]

Hi Ulrich,

I observed the same behavior when I tried using the multiple overlays and different multicast data groups for two different different AED. I guess two overlays between a pair of sites and different multicast data-groups between the AED's/Sites is not supported as of now. Another thing I observed was a weird behavior with the local ste and remote mac addresses with  this setup. When I had hosts connected between the sites and tried different fail-over scenarios the remote mac-addresses were listed as "Site local " addresses and vice-versa. It strongly made me believe that different multicast data-groups and overlays should not be used between the sites currently. I hope this will get fixed in new releases.

As far as the load-balancing or traffic forwarding is concerned that will be totally depending upon your core multicast forwarding and tree forwarding. Also at this moment AED's automatically distribute ODD and EVEN vlans between them, you cannot control it. So onfigure it and hope for the better results :-)

I would strongly recommend you to use the same multicast data-groups and Overlays to see the expected behavior.

HTH, Please rate if it does.

Cheers,

-amit singh

[Ulrich Hansen]

Hi Amit,

So, I tried to disable all Overlay-interfaces and subsequently only enabling Overlay1 on all OTV-edge devices. Theese all use the same control- and datagroup mcast-addresses (239.192.239.192 and 232.1.1.0/24 respectively). But still, the first two edge-devices form their expected local and external adjacencies, but the third and fourth still form both external and local adj with each other.

So this does not seem to be related to the use of multible overlays and multicast addresses.

/Ulrich

[Amit Singh]

Hi Ulrich,

Please could you send your topology diagram and the configurations from both the sides.

You can attach it here or send me at amisin@gmail.com

Cheers,

-amit singh

[Ulrich Hansen]

Hi Amit,

I've zipped the topology along with the vdc-configuration and Core-multicast configuration and send it to your email address.

Thanks

/Ulrich

[Ulrich Hansen]

Hi Amit,

Just wanted to let you know, that I've opened a TAC case on this issue.

/Ulrich

[Amit Singh · Accepted solution]

Many thanks for the update Ulrich, looks like that's the only to troubleshoot this further. I did not see much of the configuration issue with this. Please could you share the TAC SR number with me. I will keep any eye on this.

Cheers,

-amit singh

[Ulrich Hansen]

Hi Amit,

The case-id is SR 618224175.

I'll leave this tread open, until the problem is resolved and post the solution in here for others to see/use.

Thanks

/Ulrich

[Ulrich Hansen]

Hi Amit,

The case is closed. Apparently, the behaviour I observed are actually expected (this doesn't explain why the two OTV sites behave differently though, this could be a misconfiguration on my part). Also, according to TAC, in the upcoming Delhi-release, an OTV edge device is actually required to form both external and internal adj in order to work correctly.

Just wanted to let you know.

/Ulrich