Solved: Routing a subnet between 2 sites - Page 2

Joshua McNeely · ‎11-26-2013

Hello All,

I've been trawling through the forums here and while I've found a lot of good advice, I haven't found anything that makes sense in my case. Here's what I have:

Two of our sites are being set up with a PLC network. The vendor has decided that the only good way to accomplish cross-site monitoring is to have both sites on the same subnet. I've asked them if they would please set them to separate subnets, but they refuse claiming that this shouldn't be an issue because they've done it this way at other client sites. That said, I understand it's not really the right way to do things but they won't listen to reason.

So, both sites are residing on the same subnet of 172.X.X.Z. We do have a bit of an advantage in this situation and that is the way our sites are connected. Our switches are connected directly to each other using a telco provided ethernet connection. No MPLS, no VPN, just a direct connection. The telco equipment is completely transparent to the switches.

The switchports that are connected on each end are set up like so:

interface GigabitEthernet1/0/14

description xxxxxxx

switchport access vlan 5 (This shouldn't matter since it's been set to trunk mode)

switchport trunk encapsulation dot1q

switchport trunk native vlan 5

switchport trunk allowed vlan 5,172

switchport mode dynamic desirable

Now, if I set up a vlan interface with the 172.X.X.1 IP address on either side of the link the respective server is able to see the interface. So, the VLAN is working from the servers to the switches on both sides. However, the actual communication between the two servers (and all the PLCs on either side of the network) isn't working. It seems pretty obvious that the problem is happening somewhere on the link between the two switches but for the life of me I cannot figure out where.

Another small gotcha is that the servers are both set to use 172.X.X.1 as their default gateway. I had thought at one point that if I set up a VLAN Interface on both sides, one with .1 and one with .2 and set the servers to use those as their respective gateways that this might solve the problem. However, I'm not entirely sure if that will work if it will, whether it will require anything special or if it would be sufficient to just go

Conf t

interface vlan 172

ip address 172.X.X.1 255.255.255.0

(And similar for .2)

Thanks in advance for any help you can offer!

-Josh

Joshua McNeely · ‎11-27-2013

I take it that it wouldn't do anything if I manually changed one of the switches to be the secondary root (just now reading up on this stuff, so pardon any brain-dead questions) would it?

Jon Marshall · ‎11-27-2013

You don't want to do that. The issue is not that the link isn't working because both switches see themselves are STP root but the other way round ie. both switches see themselves as STP root because the link isn't passing vlan 172.

If you changed a switch to secondary a new root needs to be found (i think, never had the situation you have now) and it can't be the other core switch because vlan 172 is not going across that link.

The order i would do it would be to get vlan 172 going across the link. Note that when it does you will get an STP convergence anyway because you have two root bridges at the moment, so there will be a slight outage. I say slight but STP has to go across all those switches and work out potentially new paths so don't do this during key hours. Once STP has worked out which bridge is root you can then , if you are in a scheduled outage, manually set the two core switches to be root and secondary.

Like i say, i wouldn't start setting secondary roots now until the link is working for vlan 172 but i can't say for sure what would happen if you set a secondary now.

Jon

Joshua McNeely · ‎11-27-2013

Got it. Would this be a big issue for us to switch it over to trunk mode during on hours for vlan 5? See, as of right now vlan 172 can take however long it needs to sort things out and I won't mind. But I really don't want to have vlan 5 down for more than a few minutes at most. Not trying to get you to say yes or no, but I want to understand what sort of impact this is really going to have.

Thanks,
Josh

Jon Marshall · ‎11-27-2013

Josh

Firstly i'm not convinced this will solve the issue you are having because according to the "sh trunk" output they are already trunks. But i can't see what else is wrong in your config.

Because of all the switches i would not do this in key hours. I would schedule and outage to do this because there may be unforseen issues especially as i do not have a full picture of your network.

Can you post the config for the core switch in site B as well just so i can see if there is anything in there that may be wrong.

Jon

Joshua McNeely · ‎11-27-2013

Sure: A note, my boss is fairly awesome about being okay with some minor disruptions (particularly on a day prior to a holiday when people don't mind leaving a bit early) we went ahead and made the change. I'm not sure how long it would take for things to work through but so far, no luck.

Using 9731 out of 524288 bytes

!

! Last configuration change at 04:52:06 UTC Wed Nov 27 2013 by admin

! NVRAM config last updated at 04:52:09 UTC Wed Nov 27 2013 by admin

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname xxxxxxxx

!

boot-start-marker

boot-end-marker

!

enable secret 5 xxxxxxxxxxxxxx

!

username nsn-msp privilege xxxxxxxxxxxxxxx

username admin privilege xxxxxxxxxxxxxxxx

!

aaa new-model

!

aaa session-id common

switch 1 provision ws-c3750x-24

switch 2 provision ws-c3750x-24

system mtu routing 1500

ip routing

!

ip domain-name westwardseafoods.com

!

stack-power stack xxxxxxxx

mode redundant

!

stack-power switch 1

stack-power switch 2

!

crypto pki trustpoint TP-self-signed-969687808

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-969687808

revocation-check none

rsakeypair TP-self-signed-969687808

!

crypto pki certificate chain TP-self-signed-969687808

certificate self-signed 01 nvram:IOS-Self-Sig#3838.cer

!

spanning-tree mode rapid-pvst

spanning-tree portfast default

spanning-tree extend system-id

spanning-tree vlan 1-1023 priority 24576

!

port-channel load-balance src-dst-ip

!

vlan internal allocation policy ascending

!

lldp run

!

interface Port-channel1

switchport access vlan 109

switchport mode access

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

!

interface Port-channel3

switchport access vlan 109

switchport mode access

!

interface Port-channel4

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

!

interface Port-channel6

switchport access vlan 108

switchport mode access

!

interface FastEthernet0

no ip address

no ip route-cache cef

no ip route-cache

no ip mroute-cache

!

interface GigabitEthernet1/0/1

description xxxxxxxxx

switchport access vlan 109

switchport mode access

channel-group 1 mode on

!

interface GigabitEthernet1/0/2

description xxxxxxxxxxxx

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

spanning-tree portfast trunk

channel-group 2 mode on

!

interface GigabitEthernet1/0/3

description xxxxxxxxxxxxxx

switchport access vlan 109

switchport mode access

channel-group 3 mode on

!

interface GigabitEthernet1/0/4

description xxxxxxxxxxxxx

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

spanning-tree portfast trunk

channel-group 4 mode on

!

interface GigabitEthernet1/0/5

description xxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet1/0/6

description xxxxxxxxxxxxxx

switchport access vlan 108

switchport mode access

channel-group 6 mode active

!

interface GigabitEthernet1/0/7

descriptionxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet1/0/8

description xxxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet1/0/9

description xxxxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet1/0/10

!

interface GigabitEthernet1/0/11

description xxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet1/0/12

description xxxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet1/0/13

description xxxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet1/0/14

description xxxxxxxxxxxx

switchport access vlan 5

switchport mode access

!

interface GigabitEthernet1/0/15

!

interface GigabitEthernet1/0/16

!

interface GigabitEthernet1/0/17

!

interface GigabitEthernet1/0/18

description xxxxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet1/0/19

description xxxxxxxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet1/0/20

description xxxxxxxxxxx

switchport access vlan 111

!

interface GigabitEthernet1/0/21

description xxxxxxxxx

switchport access vlan 111

!

interface GigabitEthernet1/0/22

description xxxxxxxxxxxxxxx

switchport access vlan 125

switchport mode access

priority-queue out

mls qos trust dscp

!

interface GigabitEthernet1/0/23

description xxxxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet1/0/24

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 111,125,172

switchport mode trunk

priority-queue out

mls qos trust dscp

!

interface GigabitEthernet1/1/1

!

interface GigabitEthernet1/1/2

!

interface GigabitEthernet1/1/3

!

interface GigabitEthernet1/1/4

!

interface TenGigabitEthernet1/1/1

!

interface TenGigabitEthernet1/1/2

!

interface GigabitEthernet2/0/1

description xxxxxxxxxxx

switchport access vlan 109

switchport mode access

channel-group 1 mode on

!

interface GigabitEthernet2/0/2

descriptionxxxxxxxxx

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

spanning-tree portfast trunk

channel-group 2 mode on

!

interface GigabitEthernet2/0/3

description ESXi2 Port 3 vmkernel

switchport access vlan 109

switchport mode access

channel-group 3 mode on

!

interface GigabitEthernet2/0/4

description xxxxx

switchport trunk encapsulation dot1q

switchport trunk native vlan 50

switchport mode trunk

spanning-tree portfast trunk

channel-group 4 mode on

!

interface GigabitEthernet2/0/5

description xxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet2/0/6

description xxxxxxxxxx

switchport access vlan 108

switchport mode access

channel-group 6 mode active

!

interface GigabitEthernet2/0/7

!

interface GigabitEthernet2/0/8

description xxxxxxx

switchport access vlan 109

!

interface GigabitEthernet2/0/9

description xxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet2/0/10

!

interface GigabitEthernet2/0/11

!

interface GigabitEthernet2/0/12

!

interface GigabitEthernet2/0/13

!

interface GigabitEthernet2/0/14

description xxxxxxxx

switchport access vlan 5

switchport trunk encapsulation dot1q

switchport trunk native vlan 5

switchport trunk allowed vlan 5,172

switchport mode dynamic desirable

!

interface GigabitEthernet2/0/15

!

interface GigabitEthernet2/0/16

!

interface GigabitEthernet2/0/17

!

interface GigabitEthernet2/0/18

description xxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet2/0/19

description xxxxxxxxxxxxx

switchport access vlan 109

switchport mode access

!

interface GigabitEthernet2/0/20

description xxxxxxxxxxx

switchport access vlan 111

!

interface GigabitEthernet2/0/21

description xxxxxxxxxx

switchport access vlan 111

!

interface GigabitEthernet2/0/22

description xxxxxxxxxxxx

switchport access vlan 125

switchport mode access

priority-queue out

mls qos trust dscp

!

interface GigabitEthernet2/0/23

description xxxxxxxxxxx

switchport access vlan 111

switchport mode access

!

interface GigabitEthernet2/0/24

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 111,125,172

switchport mode trunk

priority-queue out

mls qos trust dscp

!

interface GigabitEthernet2/1/1

!

interface GigabitEthernet2/1/2

!

interface GigabitEthernet2/1/3

!

interface GigabitEthernet2/1/4

!

interface TenGigabitEthernet2/1/1

!

interface TenGigabitEthernet2/1/2

!

interface Vlan1

no ip address

shutdown

!

interface Vlan5

bandwidth 10000

ip address xxxxxxxxxxx 255.255.255.252

ip ospf authentication message-digest

ip ospf message-digest-key 1 md5 xxxxxxxxxxx

ip ospf cost 50

ip ospf 1 area 0.0.0.1

!

interface Vlan108

ip address xxxxxxxxxxx 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

!

interface Vlan109

ip address xxxxxxxxxxxx 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

ip ospf authentication message-digest

ip ospf message-digest-key 1 md5 xxxxxxxxxxx

ip ospf 1 area 0.0.0.1

!

interface Vlan111

ip address xxxxxxxxx 255.255.255.0

ip helper-address xxxxxxxxxx

no ip redirects

no ip unreachables

no ip proxy-arp

!

interface Vlan125

ip address xxxxxxxxx 255.255.255.0

ip helper-address xxxxxxxxxx

!

router ospf 1

router-id xxxxxxxxx

log-adjacency-changes

area 0.0.0.1 authentication message-digest

network xxxxxxxx.0 0.0.3.255 area 0.0.0.1

network xxxxxxxx.0 0.0.0.255 area 0.0.0.1

!

ip default-gateway xxxxxxxxx

ip classless

xxxxxx ip routes go here xxxxxxxxx

ip http server

ip http secure-server

!

ip access-list standard management

permit xxxxxxx

permit xxxxxxxx

!

ip sla enable reaction-alerts

snmp-server community n5nw RO management

!

banner login ^CC

CLOSED PRIVATE SYSTEM. Use subject to restriction. Authorized users only. If you have not received previous

authorization to use this system, please disconnect immediately. All access will be logged without further notice and can be

used as evidence in court.^C

!

line con 0

line vty 0 4

privilege level 15

line vty 5 15

privilege level 15

!

ntp clock-period 36027867

ntp server xxxxxxxxxxxx prefer

ntp server xxxxxxxxxxxx

end

Jon Marshall · ‎11-27-2013

Can you post "sh spanning-tree vlan 172" again.

Jon

Jon Marshall · ‎11-27-2013

Josh

Can you also post "sh vlan brief" from both core switches ?

Jon

Joshua McNeely · ‎11-27-2013

Site A

switch-dh-core#sh spanning-tree vlan 172

VLAN0172

Spanning tree enabled protocol ieee

Root ID Priority 32940

Address 081f.f3ca.6b80

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32940 (priority 32768 sys-id-ext 172)

Address 081f.f3ca.6b80

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/14 Desg FWD 19 128.14 P2p

Gi1/0/23 Desg FWD 4 128.23 P2p

Gi1/0/24 Desg FWD 4 128.24 P2p

Po1 Desg FWD 3 128.488 P2p

Po2 Desg FWD 3 128.496 P2p

Po5 Desg FWD 3 128.520 P2p

Po6 Desg FWD 3 128.528 P2p

Po13 Desg FWD 3 128.584 P2p

Gi2/0/19 Desg FWD 4 128.73 P2p

Gi2/0/23 Desg FWD 4 128.77 P2p

switch-dh-core#sh vlan brief

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

1 default active Gi1/0/9, Gi1/0/10, Gi1/0/11

Gi1/0/12, Gi1/0/15, Gi1/0/16

Gi1/0/17, Gi1/0/18, Gi1/0/19

Gi1/0/20, Gi1/0/25, Gi1/0/26

Gi1/0/27, Gi1/0/28, Gi2/0/9

Gi2/0/10, Gi2/0/11, Gi2/0/12

Gi2/0/15, Gi2/0/16, Gi2/0/17

Gi2/0/18, Gi2/0/20, Gi2/0/24

Gi2/0/25, Gi2/0/26, Gi2/0/27

Gi2/0/28

5 VLAN0005 active Gi2/0/14

104 servers active

105 management active Gi1/0/21, Gi1/0/22, Gi2/0/21

Gi2/0/22

106 storage active Gi1/0/3, Gi1/0/4, Gi2/0/3

Gi2/0/4, Po7, Po8

125 VOICE active

172 PLC active

1002 fddi-default act/unsup

1003 token-ring-default act/unsup

1004 fddinet-default act/unsup

1005 trnet-default act/unsup

Site B

switch-aly-core#sh spanning-tree vlan 172

VLAN0172

Spanning tree enabled protocol rstp

Root ID Priority 24748

Address 4055.39cc.4300

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 24748 (priority 24576 sys-id-ext 172)

Address 4055.39cc.4300

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/24 Desg FWD 4 128.24 P2p

Po2 Desg FWD 3 128.496 P2p

Po4 Desg FWD 3 128.512 P2p

Gi2/0/14 Desg FWD 19 128.68 P2p

Gi2/0/24 Desg FWD 4 128.78 P2p

switch-aly-core#sh vlan brief

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

1 default active Gi1/0/10, Gi1/0/15, Gi1/0/16

Gi1/0/17, Gi2/0/7, Gi2/0/10

Gi2/0/11, Gi2/0/12, Gi2/0/13

Gi2/0/15, Gi2/0/16, Gi2/0/17

5 VLAN0005 active Gi1/0/14

108 VLAN0108 active Po6

109 VLAN0109 active Gi1/0/5, Gi1/0/7, Gi1/0/8

Gi1/0/9, Gi1/0/13, Gi2/0/5

Gi2/0/8, Gi2/0/9, Gi2/0/18

Gi2/0/19, Po1, Po3

111 VLAN0111 active Gi1/0/11, Gi1/0/12, Gi1/0/18

Gi1/0/19, Gi1/0/20, Gi1/0/21

Gi1/0/23, Gi2/0/20, Gi2/0/21

Gi2/0/23

125 VOICE active Gi1/0/22, Gi2/0/22

172 PLC active

1002 fddi-default act/unsup

1003 token-ring-default act/unsup

1004 fddinet-default act/unsup

1005 trnet-default act/unsup

Jon Marshall · ‎11-27-2013

So both ports gi1/0/14 in site A and gi2/0/14 in site B now have this config -

int

switchport trunk encapsulation dot1q

switchport trunk native vlan 5

switchport trunk allowed vlan 5,172

switchport mode trunk

Jon

Joshua McNeely · ‎11-27-2013

Yes. So guaranteed they aren't working in Access mode.

Jon Marshall · ‎11-27-2013

So that "sh vlan brief" you posted was run before or after you updated the port config ? I ask because it is showing gi1/0/14 as being an access port in vlan 5 which it shouldn't do with the new config.

Jon

Joshua McNeely · ‎11-27-2013

It was run after, but I forgot to do a wr to write the config changes. Here's an updated set:

Site A

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

1 default active Gi1/0/9, Gi1/0/10, Gi1/0/11

Gi1/0/12, Gi1/0/15, Gi1/0/16

Gi1/0/17, Gi1/0/18, Gi1/0/19

Gi1/0/20, Gi1/0/25, Gi1/0/26

Gi1/0/27, Gi1/0/28, Gi2/0/9

Gi2/0/10, Gi2/0/11, Gi2/0/12

Gi2/0/15, Gi2/0/16, Gi2/0/17

Gi2/0/18, Gi2/0/20, Gi2/0/24

Gi2/0/25, Gi2/0/26, Gi2/0/27

Gi2/0/28

5 VLAN0005 active Gi2/0/14

104 servers active

105 management active Gi1/0/21, Gi1/0/22, Gi2/0/21

Gi2/0/22

106 storage active Gi1/0/3, Gi1/0/4, Gi2/0/3

Gi2/0/4, Po7, Po8

125 VOICE active

172 PLC active

1002 fddi-default act/unsup

1003 token-ring-default act/unsup

1004 fddinet-default act/unsup

1005 trnet-default act/unsup

Site B

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

1 default active Gi1/0/10, Gi1/0/15, Gi1/0/16

Gi1/0/17, Gi2/0/7, Gi2/0/10

Gi2/0/11, Gi2/0/12, Gi2/0/13

Gi2/0/15, Gi2/0/16, Gi2/0/17

5 VLAN0005 active Gi1/0/14

108 VLAN0108 active Po6

109 VLAN0109 active Gi1/0/5, Gi1/0/7, Gi1/0/8

Gi1/0/9, Gi1/0/13, Gi2/0/5

Gi2/0/8, Gi2/0/9, Gi2/0/18

Gi2/0/19, Po1, Po3

111 VLAN0111 active Gi1/0/11, Gi1/0/12, Gi1/0/18

Gi1/0/19, Gi1/0/20, Gi1/0/21

Gi1/0/23, Gi2/0/20, Gi2/0/21

Gi2/0/23

125 VOICE active Gi1/0/22, Gi2/0/22

172 PLC active

1002 fddi-default act/unsup

1003 token-ring-default act/unsup

1004 fddinet-default act/unsup

1005 trnet-default act/unsup

Jon Marshall · ‎11-27-2013

It's still showing gi1/0/14 and gi2/0/14 as in vlan 5 which it shouldn't because they are no longer access ports.

One other thing i noticed was that you are running STP PVST+ on switch in site A and Rapid PVST+ on switch in site B but that should not be the issue at least i would not have thought. How many switches do you have chained off the core switch in site A ie. core switch -> switch -> etc. in a chain ?

Jon

Joshua McNeely · ‎11-27-2013

In a single site I think we have maybe 3 or 4 chained together at most at this point. Possibly 5 but certainly no more than that.

So, if I'm hearing you right, vlans 5 and 172 should both not show any interfaces in the list if they are in trunk mode?

Jon Marshall · ‎11-27-2013

Josh

So, if I'm hearing you right, vlans 5 and 172 should both not show any interfaces in the list if they are in trunk mode?

Yes. I think there is something more fundmental here. Are you sure the link can carry tagged traffic ie. the telco does not have any equipment in between that could be stopping this because i cannot see any reason why vlan 172 is not allowed across the link. Vlan 5 is seen on both ends by STP but vlan 172 isn't. Vlan 5 is not tagged because it is the native vlan. Can you check with the telco that you can send tagged traffic across that link ?

Jon