Cisco Noob - Layer 3 Routing / VLAN / Spanning Tree

[Neil Kirkland]

Hi All ...

I need some pointers on which commands / settings and where, I know what I want to achieve but the things I am trying seem to be 'mutually exclusive' - either that or i'm missing something - I am not a Cisco IOS expert but I know my way around a network.

Take 3 3560 switches in Layer 3 mode, there is a 'local' fibre spanning tree ring serving mulriple switches on each, each ring is it's own IP segment / VLAN. There is then a trunk between each switch on which I want to establish a load sharing / spanning tree circuit i.e.

SW1 hosts VLAN 2 via copper on fa0/1 -12, ip address 10.10.2.254

SW1 hosts VLAN 3 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.3.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW1 hosts VLAN 10, ip address 10.10.10.1 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW2 hosts VLAN 4 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.4.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW2 hosts VLAN 10, ip address 10.10.10.2 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW3 hosts VLAN 5 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.5.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW3 hosts VLAN 10, ip address 10.10.10.3 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW1 G0/3 is a SMF trunk to SW2 G0/3

SW1 G0/4 is a SMF trunk to SW3 G0/3

SW2 G0/4 is a SMF trunk to SW3 G0/4

The trunks are configured as "trunk encapsulation dot1q", ip routing is  enabled.

I can get the trunks working OK - but I can't seem to get routing to work across them - if I define an interface on SW1 with an IP set in SW3 the switch complains so it can clearly see it so which command have I missed.

All VLAN's are part of the same domain, each VLAN has it's own DHCP hosted on it's hosting switch. The VLAN ip address is excluded from DHCP and is the default gateway for each VLAN.

All VLAN's must be able to reach VLAN2 (contains SQL servers and DNS, Time etc etc), the VLAN's are working, DHCP etc is all working - but I can't get anything other than VLAN 10 IP's to talk across the trunks - I've tried adding spanning-tree vlan 2,3,4,5,10 but this hasn't worked, the ip route-map shows nothing, if you show spanning-tree the trunk ports do show up as an interface for all VLAN's - and yet no traffic passes across them - show route displays nothing. I tried adding ip route 10.10.*.0 255.255.255.0 10.10.2.254 (where 10.10.2.254 is the ip address of VLAN 2) but that's done nothing.

I have tried various combinations - unsuccessful so far - I need the trunks to be not only fault tolerant but load sharing which kind of negates fixing IP's on them - or does it ?? - what am I missing ?

(switches are all running IP services IOS)

[Jon Marshall]

Neil

So vlan 10 is used for simply connecting the switches together ie. no clients in that vlan ?

If so there are a couple of ways of doing it but trunks are not the answer unless you make one 3560 switch responsible for all inter vlan routing whereas you are doing the routing locally on each switch.

I think routing between the switches is the best solution here. You can use static routes or if you are happy to you could use EIGRP or OSPF. Each switch would see direct routes to the other switch's subnets and if the direct link failed then the indirect link could be used. A routing protocol would be better at this than using static routes as you would need to use floating statics in this scenario.

So i can help with the config but if you could just confirm -

1) is vlan 10 only used for interconnectivty

2)  are you happy to run a routing protocol and any preference eg. EIGRP/OSPF (probably not RIP )

Jon

[JohnTylerPearce]

I agree with Jon Marshall. From the looks of this setup, I would implement EIGRP, and either configure 'variance' if you have too, and or some other metric to implement load-balancing of links.

[Jon Marshall]

John

It's a good point and needs clarifying but i don;t think variance or anything else is need for load balancing because each switch will use it's direct link to get the subnets on the other switch.

Neil, perhaps you can clarify. Do you want one switch to be able to use both links to get to one subnet on one of the other switches or are you happy if the destination subnet is on one switch only the link to that switch is used. I'm assuming the server subnet is the key subnet in terms of client access.

If you do want to use both links from each switch to the server subnet then the options are -

1)  don't have a mesh between switches but use etherchannel trunks to the switch with the server subnet and move all the inter vlan routing to the server subnet switch

or

2) again no mesh but use L3 etherchannels from each switch to the server subnet switch. The routing for vlans would stay local to each switch.

both would mean if the server subnet switch went down you have lost connectivity between the other two switches but this may not be an issue for you.

3) have a mesh and use variance as John suggested. This would still mean one link was used more than the other and people have reported issues with using it on these forums although i have never used it so can't say for sure. I can't see a reason why it wouldn't work.

Perhaps you could clarify what the main goal is ?

Edit - if you do choose one of the above as opposed to the original suggestion then if you choose 1) no routing protocol needed because all L3 SVIs on same switch. If you choose 2) then choice between EIGRP/OSPF and if you choose 3) then EIGRP it is.

Jon

[Neil Kirkland]

Thanks for the input guys, yes VLAN 10 is for nothing more than the interconnects although I'm not sure I even need it.

There is one 'root' which is SW1 hosting the servers, in reality I think that SW2 will go direct to SW1 and SW3 direct to SW1. The remaining 'trunk' lighting up only on failure of the other or a satellite switch and at times of high demand (although this is simply an idea not a necessity but it would be nice to share the load if possible). I know that the root is still a single point of failure but dual power supplies, feeds will hopefully aleviate most risk - can't remember when the last switch failed totally.

The main goal is to isolate 3 buildings in terms of broadcast and security, to also provide some resilience for the 'local' network should one building go 'ape' although all switches etc are UPS and generator supported.

Buildings two and three are 'un manned' mostly, building one hosts the data warehouse to which Building 1, 2 and 3 send data and also the control centre - the servers are isolate on vlan 2 which will also serve as the 'management' vlan.

The server subnet should be accessible but controllable (I semi planned to implement ACL's at some point based on MAC address - but this is future).

Sorry I'm rambling a bit -- done way too many hours this week - hope I'm making sense - I'll explore EIGRP whilst I await a response, I am happy to use a routing protocol and appriciate the support / input.

[Jon Marshall]

Neil

Okay, if you are happy to use a full mesh and using the direct link from SW2 and SW3 to SW1 as the main route then -

SW1

====

int gi0/3 <-- this connects to SW2

no switchport

ip address 10.10.10.1 255.255.255.252

int gi0/4 <--- this connects to SW3

no switchport

ip address 10.10.10.5 255.255.255.252

SW2

====

int gi0/3 <--- this connects to SW1

no switchport

ip address 10.10.10.2 255.255.255.252

int gi0/4 <-- this connects to SW3

no switchport

ip address 10.10.10.9 255.255.255.252

SW3

====

int gi0/3  <-- this connects to SW1

no switchport

ip address 10.10.10.6 255.255.255.252

int gi0/4 <-- this connects to SW2 

no switchport

ip address 10.10.10.10 255.255.255.252

so with the above each switch has a L3 routed connection to each of the other switches.

For EIGRP you need to then add the above networks plus each switches local subnets so -

SW1

====

router eigrp 10

network 10.10.10.0 0.0.0.7

network   note if the vlans are /24s then the wildcard is 0.0.0.255

network

no auto-summary

if the vlans are not using /24s and you need help with the wildcards just let me know.

SW2

====

router eigrp 10

network 10.10.10.0 0.0.0.3

network 10.10.10.8 0.0.0.3

network

no auto-summary

SW3

====

router eigrp 10

network 10.10.10.4 0.0.0.3

network 10.10.10.8 0.0.0.3

network

no auto-summary

after you have done the above you should then be able to do a "sh ip route" on each switch and see routes to the other subnets on the other switches. As i say they will only use their direct links but if for example SW2 lost it's connection to SW1 it should then see routes to the subnets on SW1 via SW3.

Obviously it goes without saying there will be downtime while you set this up.

Let me know if you need anything clarifying.

Jon

[Neil Kirkland]

WOW - what can I say  many thanks for this. I'm only just getting started and it takes time to take all this in, I link to understand what I'm doing, why things work (and sometimes don't). The vlans are using /24 (with an exclusion for the root IP) . I have a subnet for servers (VLAN2), subnet for 'devices / users', VLAN3, 4 and 5 in buildings 1 to 3 respective and a subnet for 'switch management' VLAN10 - aince I was advised to keep the switch subnet isolated from everything else.

I'll put this info to good use and try to build on it.

I've played with switches before but not Ciscos - but never to this depth - currently lining myself up for some Cisco exams but there is so much going on in the IOS - it's like learning the Microsoft ,NET libraries and classes - spend a hundred hours building a solution only to have somebody show you a two liner that does the same in 10 minutes ...

Really approciate this ... guess I'll be spending a lot of time here.

[Jon Marshall]

Neil

No problem, glad to have helped.

Any issues with it don't hesitate to come back and yes, these forums are a great place to get answers to specific problems but also to clarify any specific concepts etc. that you may be struggling with.

Jon

[Neil Kirkland]

I'm currently playing with it, figured out all the masking etc - I think ... interesting that you restricted each interswitch link to a single subnet. Trying to work out the reasoning by playing and reading the books ...

Right now the SW2 is being stubborn and refusing to come up - sh ip route shows only the directly connected channels - the local vlan subnet doesn't appear - erased and started again a few times - plus the 'local' copper channels will not enable so I can't access vlan 4 locally to prod it ..

I need to do a bit more reading on this eigrp routing to figure out specifics on it - i.e. a route is up between SW1 and SW3 but you can't ping the DNS server on VLAN 2 from SW - and you should be able to -

Having fun as they say

[Jon Marshall]

Neil

Can you post "sh ip route" from SW1 and SW3 and specify the client IP address you are pinging from and the DNS sever IP.

The reason for using a single subnet is because these are L3 routed links so each link needs a dedicated IP subnet for it.

It can help to think of each switch as a router in this regard and you have simply created a L3 triangle between all routers.

Jon

[Neil Kirkland]

Hi John ,, here is the sh ip route and sh ip eigrp from all three.

The ip address I'm trying to reach from SW1, SW2 is 10.10.2.253 - the DNS server - the server is available and connected to a copper port designated and assigned to VLAN 2 (which has the root ip of 10.10.2.254) dhcp is not enabled for VLAN 2.

I can ping the DNS box from VLAN 5 (same switch as VLAN 2).

The copper ports on the SW1 and SW2 boxes refuse to 'come up' - they remain shutdown no matter what. I haven't yet configured VLAN 10 ....

(NOTE - these switches are on the bench right now - I intend to ge tthe config sorted / tested and verified before they go into production)

SWITCH 1 - Host for VLAN 3 and 10

SW1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
D       10.10.2.0/24 [90/3072] via 10.10.10.6, 01:19:29, GigabitEthernet0/2
C       10.10.10.0/30 is directly connected, GigabitEthernet0/1
C       10.10.10.4/30 is directly connected, GigabitEthernet0/2

SW1#sh ip eigrp interfaces
EIGRP-IPv4:(10) interfaces for process 10

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Gi0/2              1        0/0         1       0/1            0           0
Vl3                0        0/0         0       0/1            0           0
SW1#

SWITCH 2 - Host for VLAN 4 and 10

SW2#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/30 is subnetted, 2 subnets
C       10.10.10.8 is directly connected, GigabitEthernet0/1
C       10.10.10.0 is directly connected, GigabitEthernet0/2

SW2#sh ip eigrp interfaces
EIGRP-IPv4:(10) interfaces for process 10

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Gi0/2              0        0/0         0       0/1            0           0
Gi0/1              0        0/0         0       0/1            0           0
Vl4                0        0/0         0       0/1            0           0
SW2#

SWITCH 3 - Host for VLAN 2, 5 and 10

SW3#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C       10.10.10.8/30 is directly connected, GigabitEthernet0/1
C       10.10.2.0/24 is directly connected, Vlan2
C       10.10.10.4/30 is directly connected, GigabitEthernet0/2
SW3#sh ip eigrp interfaces
EIGRP-IPv4:(5) interfaces for process 5

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Vl2                0        0/0         0       0/1            0           0
Vl5                0        0/0         0       0/1            0           0
EIGRP-IPv4(0)(0) interfaces for process 0

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
EIGRP-IPv4:(10) interfaces for process 10

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Gi0/2              1        0/0         1       0/1           50           0
Vl5                0        0/0         0       0/1            0           0
Vl2                0        0/0         0       0/1            0           0
SW3#

SW3#show vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active   
2    SERVERS                          active    Fa0/13, Fa0/14, Fa0/15, Fa0/16
                                                Fa0/17, Fa0/18, Fa0/19, Fa0/20
4    DB5-LAN                          active    Fa0/1, Fa0/2, Fa0/3, Fa0/4
                                                Fa0/5, Fa0/6, Fa0/7, Fa0/8
                                                Fa0/9, Fa0/10, Fa0/11, Fa0/12
                                                Gi0/1, Gi/2

10   MANAGER                          active    Fa0/21, Fa0/22, Fa0/23, Fa0/24
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0  
2    enet  100002     1500  -      -      -        -    -        0      0  
3    enet  100003     1500  -      -      -        -    -        0      0  
4    enet  100004     1500  -      -      -        -    -        0      0  
5    enet  100005     1500  -      -      -        -    -        0      0  
10   enet  100010     1500  -      -      -        -    -        0      0  
1002 fddi  101002     1500  -      -      -        -    -        0      0  
1003 tr    101003     1500  -      -      -        -    srb      0      0  
1004 fdnet 101004     1500  -      -      -        ieee -        0      0  
1005 trnet 101005     1500  -      -      -        ibm  -        0      0  
         
Remote SPAN VLANs
------------------------------------------------------------------------------
         

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

PPS : I'm using ports Gi0/1 and Gi0/2 for now - I removed these from DB5-LAN and can now 'ping' from SW1 but not from SW2 - but the local copper is still dead on SW1 and SW2

Copper channels not dead - faulty patch lead ... the simplest things ....

[Jon Marshall]

Neil

Okay, there are a few routes missing from EIGRP so i suspect they are missing from the config.

Can you do me a favour and post the full configuration of each switch ie. a "sh run" from each switch.

Can you post the configs as attachments if possible to keep the thread manageable. Note to add attachments when click on reply there is on the top right of the message box a "Use advanced editor" link. Click on that and then you can add attachments as well as post a response.

Finally, you have vlan 10 with ports in it on SW3. I thought vlan 10 was only to be used for connecting the switches and because we are now using L3 links we shouldn't need vlan 10.

So what are those ports in vlan 10 for ?

Jon

[Neil Kirkland]

Just blew the eigrp away and am rebuilding - good practice  ... will post config once done

The vlan 10 ports are a legacy thing - just not got rid - I was using them as a span to monitor traffic and also add management access - i.e keep the switch management LAN segregated from all others. I'll be an hour or two - got to collect the other half shortly ... she thinks valentines day is important - well no harm thinking is there

I've got switches to play with

[Jon Marshall]

Neil

she thinks valentines day is important - well no harm thinking is there

Nope, that is definitely more important than this

Couple of things to note -

1) if you need to SPAN from one switch to another you can't do that using L3 routed links. You can with ERSPAN but the 3560s don't support ERSPAN.

If this is going to be a big problem then we need to modify the config. Not hugely but it would need modifying.

2) keeping the management of switches separate is a good idea. Again though with L3 routed links you can't have a common vlan between all switches. So to manage the switches in this setup you would use loopback addresses instead of L3 vlan interfaces.

Both 1) and 2) using a vlan for management as opposed to loopbacks can be accomodated with a few config changes but how important are they to you ?

Jon

[Neil Kirkland]

Sorry for delay - file is attached.

Don't care over much about SPAN across the links I can always set up a local laptop / span in the rack.

I do care about being able to manage in a secure manner without walking a mile and a half

I also want to make sure that I still have access should 'storm's etc take over a LAN ... and keep the IP address of the switch away from all VLAN 3, 4 and 5 users, in fact I would like to adopt ACL's on the switch management and SQL server ports I'd like to stick the SQL behind a firewall internally but the accountants won't play nice.

I've edited out all the fa0/ ports on the reports