Layer 2 Trunk over ISP MPLS...

Venture101 · ‎05-05-2011

Hello,

I recently was asked to turn a routed link between our HQ and our DR Site into a trunked link to allow us to span our main Server VLAN up to the DR Site.

I was informed by the ISP that owns the 100Mb Leased Line between the sites that the link was configured to pass dot1q traffic and I should just have to configure my links at either end as trunks to get the link to come up.

There is a Cisco 3560 at either end with Layer 3 routing enabled (obviously as this was a routed link previously).

So, I scheduled an outage and configured either end as a trunk links as follows and repointed any routes from the /30 routed addresses to the management addresses of the switches on either side:

***For info VLAN 15 is the DR Site and VLAN 11 is the Server range at our HQ and was available on the connected switch***

interface FastEthernet0/24

switchport trunk encapsulation dot1q

switchport trunk native vlan 15

switchport trunk allowed vlan 11,15

switchport mode trunk

switchport nonegotiate

speed 100

duplex full

Basically this did not work at all. I could see no traffic coming over the links, no layer 2 but the trunks were showing as up between my switches and the ISPs equipment.

On my 3rd attempt of trying to get this working I created an IP Address for VLAN 15 on our HQ Switch on the off-chance this would bring up all routing to the site and it worked!

Am I going crazy? If I have trunked a VLAN down from the DR site onto our network (VTP is configured so once I configured it on the core network the VLAN existed all over the HQ site) I shouldnt have to create an instance of that IP Address to get the link working?

See below configs of the working configs. The only thing changed to get this working was the creation of an IP Address on VLAN 15 which was already showing as active on the switch before I created this IP Address????

Have removed sensitive information for security purposes.

Obviously this is working now but I just want confirmation for my sanity that I should have had to create that address at the HQ site. I think there is an issue with the ISPs config and want to push it back to them if possible.

HQ CONFIG

!
! Last configuration change at 12:44:00 BST Mon May 2 2011 by abz-admin
! NVRAM config last updated at 13:08:00 BST Mon May 2 2011 by abz-admin
!
version 12.2
service nagle
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
service password-encryption
service sequence-numbers
no service dhcp
!
##############################################################
!
no logging console
##############################################################
!
##############################################################
no aaa new-model
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
system mtu routing 1500
ip subnet-zero
no ip source-route
ip routing
no ip domain-lookup
!
!
mls qos
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
class-map match-all JH-QoS
match access-group 150
!
!
policy-map CEU-JH-QoS
class JH-QoS
    police 419424000 8000 exceed-action drop
class class-default
    police 419424000 8000 exceed-action drop
!
!
!
interface FastEthernet0/1
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface FastEthernet0/2
shutdown
!
interface FastEthernet0/3
shutdown
!
interface FastEthernet0/4
shutdown
!
interface FastEthernet0/5
shutdown
!
interface FastEthernet0/6
shutdown
!
interface FastEthernet0/7
shutdown
!
interface FastEthernet0/8
shutdown
!
interface FastEthernet0/9
shutdown
!
interface FastEthernet0/10
switchport access vlan 50
switchport mode access
!
interface FastEthernet0/11
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/12
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/13
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/14
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/15
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/16
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/17
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/18
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/19
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/20
switchport access vlan 50
switchport mode access
shutdown
!
interface FastEthernet0/21
shutdown
!
interface FastEthernet0/22
shutdown
!
interface FastEthernet0/23
!
interface FastEthernet0/24
##############################################################
switchport trunk encapsulation dot1q
switchport trunk native vlan 15
switchport trunk allowed vlan 11,15
switchport mode trunk
switchport nonegotiate
logging event trunk-status
logging event bundle-status
speed 100
duplex full
!
interface GigabitEthernet0/1
shutdown
!
interface GigabitEthernet0/2
shutdown
!
interface Vlan1
description Venture Kings Close LAN
ip address x.x.x.x 255.255.255.0
no ip redirects
no ip proxy-arp
!
interface Vlan15
ip address x.x.x.x 255.255.255.0
!
interface Vlan50
ip address x.x.x.x 255.255.255.0
!
router eigrp 1
redistribute static route-map STATIC
passive-interface default
no passive-interface Vlan1
network x.x.x.x
no auto-summary
!
ip classless
ip route x.x.x.x 255.255.255.192 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip http server
!
!
access-list 1 permit x.x.x.x 0.0.0.255
access-list 15 remark VTY access
access-list 15 permit x.x.x.x 0.0.255.255
access-list 15 permit x.x.x.x 0.0.0.255
access-list 15 permit x.x.x.x 0.0.0.255
access-list 150 permit ip x.x.x.x 0.0.0.63 any
access-list 150 permit ip any x.x.x.x 0.0.0.63
route-map STATIC permit 10
match ip address 1
!
##############################################################
!
control-plane
!
banner exec
******************************************************************************
*           LOCATION:               *
********************************************************************

!
line con 0
##############################################################
login
line vty 0 4
##############################################################
login local
length 0
transport input telnet
line vty 5 15
##############################################################
login local
transport input telnet
!
ntp clock-period 36029047
ntp server x.x.x.x
ntp server x.x.x.x
ntp server v
end

DR CONFIG

!
! Last configuration change at 14:53:07 GMT Wed May 4 2011 by abz-admin
! NVRAM config last updated at 14:53:10 GMT Wed May 4 2011 by abz-admin
!
version 12.2
service nagle
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
service password-encryption
service sequence-numbers
no service dhcp
!
############################################################
!
no logging console
############################################################
!
############################################################
no aaa new-model
clock timezone GMT 0
system mtu routing 1500
ip subnet-zero
no ip source-route
ip routing
no ip domain-lookup
############################################################
ip name-server x.x.x.x
!
ip tftp source-interface Vlan15
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
class-map match-all JH-QoS
match access-group 150
!
!
policy-map CEU-JH-QoS
class JH-QoS
police 419424000 8000 exceed-action drop
class class-default
police 419424000 8000 exceed-action drop
!
!
!
interface Port-channel1
switchport access vlan 15
switchport mode access
!
interface FastEthernet0/1
############################################################
switchport access vlan 15
switchport mode access
switchport nonegotiate
!
interface FastEthernet0/2
switchport access vlan 15
switchport mode access
!
interface FastEthernet0/3
switchport access vlan 15
switchport mode access
!
interface FastEthernet0/4
switchport access vlan 15
switchport mode access
!
interface FastEthernet0/5
switchport access vlan 15
!
interface FastEthernet0/6
switchport access vlan 15
shutdown
!
interface FastEthernet0/7
switchport access vlan 15
shutdown
!
interface FastEthernet0/8
switchport access vlan 15
shutdown
!
interface FastEthernet0/9
switchport access vlan 15
shutdown
!
interface FastEthernet0/10
switchport access vlan 15
shutdown
!
interface FastEthernet0/11
switchport access vlan 15
shutdown
!
interface FastEthernet0/12
switchport access vlan 15
shutdown
!
interface FastEthernet0/13
switchport access vlan 15
shutdown
!
interface FastEthernet0/14
switchport access vlan 15
shutdown
!
interface FastEthernet0/15
switchport access vlan 15
shutdown
!
interface FastEthernet0/16
switchport access vlan 15
shutdown
!
interface FastEthernet0/17
switchport access vlan 15
shutdown
!
interface FastEthernet0/18
switchport access vlan 15
shutdown
!
interface FastEthernet0/19
switchport access vlan 15
shutdown
!
interface FastEthernet0/20
switchport access vlan 15
shutdown
!
interface FastEthernet0/21
no switchport
no ip address
!
interface FastEthernet0/22
############################################################
switchport access vlan 15
switchport mode access
shutdown
speed 100
duplex full
no cdp enable
!
interface FastEthernet0/23
switchport access vlan 15
switchport mode access
!
interface FastEthernet0/24
############################################################
switchport trunk encapsulation dot1q
switchport trunk native vlan 15
switchport trunk allowed vlan 11,15
switchport mode trunk
switchport nonegotiate
logging event trunk-status
logging event bundle-status
speed 100
duplex full
!
interface GigabitEthernet0/1
############################################################
switchport access vlan 15
switchport mode access
channel-group 1 mode active
!
interface GigabitEthernet0/2
############################################################
switchport access vlan 15
switchport mode access
channel-group 1 mode active
!
interface Vlan1
############################################################
no ip address
no ip redirects
!
interface Vlan7
############################################################
ip address x.x.x.x. 255.255.255.248
!
interface Vlan15
############################################################
ip address x.x.x.x 255.255.255.0
no ip redirects
!
interface Vlan215
############################################################
ip address x.x.x.x 255.255.255.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 x.x.x.x name ASA-FW
ip route x.x.x.x 255.0.0.0 x.x.x.x
ip route x.x.x.x 255.255.255.192 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.0.0 x.x.x.x
ip route x.x.x.x 255.255.255.252 x.x.x.x
ip route x.x.x.x 255.255.255.252 x.x.x.x
ip route x.x.x.x 255.255.255.0 x.x.x.x
ip route x.x.x.x 255.255.255.252 x.x.x.x
ip http server
!
!
access-list 15 permit x.x.x.x 0.0.255.255
access-list 150 permit ip x.x.x.x 0.0.0.63 any
access-list 150 permit ip any x.x.x.x 0.0.0.63
snmp-server community public RO
####################################
!
control-plane
!
!
line con 0
####################################
login local
line vty 0 4
####################################
login local
length 0
transport input telnet
line vty 5 15
####################################
login local
transport input telnet
!
####################################
end

marcusbrutus · ‎05-05-2011

Hi Ewan,

Am not sure if i exactly understand your setup. You configured vlan 15 as your native vlan on the concerned interface which means that your provider doesn't expect any tagging going through their link. However i am inclined to believe that vtp information between HQ and DR should work. So the trunk link should be up. But not specifying an IP on vlan 15 would prevent any routing from the other vlans to vlan 15 and vice versa.

Venture101 · ‎05-05-2011

Hi Marcus,

I had tried this setup with no native VLAN and the trunk link still did not come up. By the way when I say if didnt come up I mean that I couldnt ping from VLAN15 to any of the addresses on the HQ network.I could schedule to take out the native vlan command and test the results.

My understanding was if I have tagged those VLANs across the link and created VLAN15 on the HQ network should I not be able to ping across that trunked link into the HQ network from VLAN15 at the DR site? Can you confirm?

Its potentially just the Layer 3 aspect enabled on these switches which is throwing me off I've never setup this sort of link over a leased line before its usually always a routed link. Another complication is that I still required the DR Switch to be the default gateway for this site as it has its owned MPLS/Internet connections etc.

Every days a school day

Thanks

Ewan

Venture101 · ‎05-06-2011

For info I had to roll this change back to being a routed link there was something definitely not right with the link as backups over it and other services were failing.

Back to the drawing board.

marcusbrutus · ‎05-07-2011

Hi Ewan,

Sorry for the late reply; I was out for a couple of days.

From an MPLS/leased line perspective, it should be pretty straightforward for them since they just deal with L2 and all they really care about is the reachability of their PE/CE.

If the L2 interface is up but icmp is still failing, then it is normally an L3 problem.

With native vlan 15 specified, ping from the DR will have a source IP of the vlan 15 interface. This packet will not be tagged. When it reaches the HQ switch, since it is not control type traffic and is instead an ICMP packet, it will check the routing table for the destination IP (L3). If the destination IP is on an interface on vlan 15 where there is no int vlan 15 IP then this packet will be dropped. If the destination is on another vlan then packets in reply will also be dropped since there is no vlan 15 routing entry.

The native vlan i believe is only significant to the line provider; if they want traffic from your network to be tagged or untagged. Essentially, all vlans should have IPs specified unless you wish to contain all traffic to just one vlan.

Hope this helps.

Venture101 · ‎05-08-2011

Great thanks Marcus,

Tomorrow morning Im going to try and set this up again but Im going to try and simplify things by removing the IP Routing from the 2 switches involved in the connection so they are only dealing with L2.

I can move the routing for the DR Site to the firewall onsite.

Will let you know how I get on and thanks.

Ewan

Venture101 · ‎05-09-2011

So another update.

Today I basically tried creating the trunk link again today but what I did was create VLAN15 as I normally would any other VLAN. An IP Address on each of my cores and then an HSRP Address for it.

When I trunked the Link over my switch at HQ I took all its Layer 3 capability away so it was just L2.

At the DR Site I set the trunk up and pointed all routes towards the HSRP Address of my cores.

Everything came up fine but within a few minutes the data replications between the sites were failing. This is bizarre because no connectivity is being lost between the sites at all. I think there is potentially some weird Layer 2 problem going on that is screwing up the replications.

Will keep investigating and let you know once resolved.