ARP issue?

[damige1985]

Hello Gentlemen,

Our company is experiancing difficulties connecting a (Stretched Layer2) Dot1q trunk between DC and DR sites.

Our current setup (which works without problems) is a basic Dot1q trunk (on both sides) running a vlan allowed list and filtering BPDU.

Our future setup is running 2 links between both sites. The setup on both sides of link1 is as follows:

switchport trunk encapsulation dot1q

switchport mode trunk

bandwidth 400000

spanning-tree mst 0-3 cost 1000

spanning-tree mst 4 cost 1000000

Link 2:

switchport trunk encapsulation dot1q

switchport mode trunk

bandwidth 400000

spanning-tree mst 0-3 cost 1000000

spanning-tree mst 4 cost 1000

I can see spanning-tree MST work without issues, discarding ports where expected and forwarding on others. I have also checked the logging and there is currently no spanning-tree loop with this setup.

This is where the problem starts. A (what seems) random amount of hosts will be able to connect intra-vlan(so e.g vlan 5 to vlan 5) across this link but others are not able to communicate. When diagnosing this issue i see that when site2 sends out an arp, site 1 receives this and responds.(checked this by wiresharking the link on site 1) But somehow the address does not show up in the arp table on any device in site 2.

• We have tried rebooting the site 2 switch (made no difference whatsever)
• We have also tried "shut no shut" on the interface in site1. The effect of this was strange since; somehow the site 2 arp table got a different set of addresses running this time. And stopped responding to arps that had been working before the "shut no shut".
• We have also tried filtering BPDU's ( so it becomes seperate MST domains )
• Running it as a single link.
• Moving the link to a different switch

All the same problems...

This problem even shows on a simpel L2 vlan, with no gateway. Site2 simply does not seem to be able to fill its arp tables regardless of the type of host used (switches, windows servers)

As soon as we switch back to the old link. everything works just fine.  All arps come trough without exceptions.

This to me sounds alot like a bug. Or maybe a provider issue ?

Switches are all 3750 stacked running 12.2(55)SE1, VTP pruning disabled, MST areas match on all switches.

Please let me know what you think

Best regards,

Paul

[Giuseppe Larosa]

Hello Paul,

>>We have also tried filtering BPDU's ( so it becomes seperate MST domains )

No, this is not correct. You need MST BPDUs to flow on the two links to build loop free topologies for each MST instance so remove any form of STP BDPU filtering.

The use of STP BPDU filtering is not compatible with a topology with multiple links between the two sites.

Edit:

it can also be a provider related issue if their EoMPLS or VPLS service is actually not supporting broadcast or multicast or unknown unicast traffic as ARP requests are carried within ethernet broadcast frames.

Check with

show spanning-tree interface type x/y

command that both sides of the same link agree on the Designated bridge ID of the link to verify effective communication with MST BPDUs.

Hope to help

Giuseppe

[damige1985]

Guiseppe,

You are correct in the fact that it is not advisable to block STP when using multple links.

Just to clarify:

We only used BPDU filter in two scenarios:

• Single link between DC-DR (no stp loops this way)
• When using Trunk allow lists to split vlan traffic (keeping in mind to change the native vlan)

In our final setup the configuration was as shown in the initial post (so no bpdu filter's applied)

Thanks for your answer though,

Edit:

"it can also be a provider related issue if their EoMPLS or VPLS service is actually not supporting broadcast or multicast or unknown unicast traffic as ARP requests are carried within ethernet broadcast frames."

Im am personally not very familiar with EoMPLS or VPLS ( on a ccna level with these specific topics ) Can this clarify the seemingly "random" behaviour though? To me it seems an "all or nothing" sitaution would be the case if its an issue with sending encapsulated broadcasts.

Edit2:

Check with

show spanning-tree interface type x/y

command that both sides of the same link agree on the Designated bridge ID of the link to verify effective communication with MST BPDUs.

We checked the STP configuration and they show up as expected. MST 0-3 discarding on the one link, MST 4 discarding on the other.

Also BPDU counters show two-way communication.

Paul

[Giuseppe Larosa · Accepted solution]

Hello Paul,

you should not use BDPU filtering even if using manually  configured trunks as MST BPDU are actually sent once with MST sections  inside IST instance 0 BPDUs.

Are the two  switch endpoints able to exchange MST BPDUs on the links?

I see the answer is positive. This is good news.

The STP BPDUs are an example of traffic with multicast  destination, if the issues is on the service provider side I would  expect issues also for STP BPDUs exchange.

I would ask the SP support for this issue, as it looks like intermittent connectivity over the service.

Hope to help

Giuseppe

[damige1985]

Hello Guiseppe,

We have contacted the ISP and they informed us that they limit the links to 200mac-addresses.

This solves this issue we had, but gives us a new problem.

How to limit our mac-adresses.

• NAT overload is not an option
• nor is making the interfaces routed since the vlan is a stretched L2 /22 network.
• Tunneling wont work since the vlan is directly connected

Any tips?

[Giuseppe Larosa · Accepted solution]

Hello Paul,

>>We have contacted the ISP and they informed us that they limit the links to 200mac-addresses.

This explains the intermittent behaviour.

The most viable option would be to ask them for an increase to 1,000 Mac addresses for an additonal (but reasonable) fee.

As an alternative you should deploy routers with L2TPv3 capability between the links, but be aware that performance would be limited by router's performances.

see

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t2/feature/guide/gtl2tpv3.html

This would be a form of point to point L2 tunneling over an IP tunnel and would allow to overcome the MAC address limits.

L2TPv3 would require Vlan based sub interfaces and can be selective.

For full GE speeds a pair of ASR 1000 would be needed so it may be expensive.

Hope to help

Giuseppe

[damige1985]

Thanks Guiseppe