Cisco Noob - Layer 3 Routing / VLAN / Spanning Tree

[Neil Kirkland]

Hi All ...

I need some pointers on which commands / settings and where, I know what I want to achieve but the things I am trying seem to be 'mutually exclusive' - either that or i'm missing something - I am not a Cisco IOS expert but I know my way around a network.

Take 3 3560 switches in Layer 3 mode, there is a 'local' fibre spanning tree ring serving mulriple switches on each, each ring is it's own IP segment / VLAN. There is then a trunk between each switch on which I want to establish a load sharing / spanning tree circuit i.e.

SW1 hosts VLAN 2 via copper on fa0/1 -12, ip address 10.10.2.254

SW1 hosts VLAN 3 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.3.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW1 hosts VLAN 10, ip address 10.10.10.1 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW2 hosts VLAN 4 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.4.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW2 hosts VLAN 10, ip address 10.10.10.2 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW3 hosts VLAN 5 via a fibre spanning tree circuit on G0/1 & G0/2, dhcp 10.10.5.0/24, trunk 1 on G0/3 and trunk 2 on G0/4

SW3 hosts VLAN 10, ip address 10.10.10.3 (trunks 1 and 2 have no IP address but are members of VLAN 10)

SW1 G0/3 is a SMF trunk to SW2 G0/3

SW1 G0/4 is a SMF trunk to SW3 G0/3

SW2 G0/4 is a SMF trunk to SW3 G0/4

The trunks are configured as "trunk encapsulation dot1q", ip routing is  enabled.

I can get the trunks working OK - but I can't seem to get routing to work across them - if I define an interface on SW1 with an IP set in SW3 the switch complains so it can clearly see it so which command have I missed.

All VLAN's are part of the same domain, each VLAN has it's own DHCP hosted on it's hosting switch. The VLAN ip address is excluded from DHCP and is the default gateway for each VLAN.

All VLAN's must be able to reach VLAN2 (contains SQL servers and DNS, Time etc etc), the VLAN's are working, DHCP etc is all working - but I can't get anything other than VLAN 10 IP's to talk across the trunks - I've tried adding spanning-tree vlan 2,3,4,5,10 but this hasn't worked, the ip route-map shows nothing, if you show spanning-tree the trunk ports do show up as an interface for all VLAN's - and yet no traffic passes across them - show route displays nothing. I tried adding ip route 10.10.*.0 255.255.255.0 10.10.2.254 (where 10.10.2.254 is the ip address of VLAN 2) but that's done nothing.

I have tried various combinations - unsuccessful so far - I need the trunks to be not only fault tolerant but load sharing which kind of negates fixing IP's on them - or does it ?? - what am I missing ?

(switches are all running IP services IOS)

[Jon Marshall]

Neil

Managing them won'tt be any different than using a common vlan you just use loopbacks so once we get the connectivity sorted we can add loopbacks to each switch and advertise them into EIGRP.

In terms of the configs -

1) can you on each switch -

switch(config)# ip subnet-zero

2) then we'll concentrate on SW1 and SW3. Firstly be aware that a L3 vlan interface only comes up if there is a port in that vlan that is up/up ie. you need a connected device in that vlan. Without it, it will be down.

So on both SW1 and SW3 can you do a "sh ip int br" and check

a) the status of the physical ports that connnect SW1 to SW3

b) the status of the L3 vlan interface. If the vlan interface is showing down/down then -

int vlan

no shut

then check again. Like i say you need a device connected to the switch in that vlan as well.

If all interfaces are showing as up/up and you are still not seeing routes we will need to look at the EIGRP neighborships.

Jon

[Neil Kirkland]

Really grateful for the assist, nothing on the vlans - I'll add some copper ports to them and plug something in - as I said these are on the bench right now ..

How come we need to allow subnet-zero - I've laways regarded such things as a bad idea - no matter it shall be done.

I've already configured the VLAN interface for no shut but clearly the switch ignores that if there is no connection.

Long time since I felt like a beginner but I sure do right now .. but thats because I am in Cisco terms

[Jon Marshall]

Neil

No problem with the help, we've all been there learning about Cisco ways of doing things and still are

ip subnet-zero is an old hangover and it just isn't needed nowadays ie. the default on Cisco devices is now "ip subnet-zero".

In terms of the SVI (vlan interface) being down if you had used trunks between the switches that would have been enough to bring them up but as they are L3 links they won't do this.

I suspect by now you are wondering why we are using  L3 links and not trunks but the routed solution is the best design for your setup, it just makes testing a little bit harder.

If getting a connected device on each switch is going to be problematic we can use loopbacks (as they are always up)  on each switch to test EIGRP routing and connectivity between the switches just so you can see how it is going to work.

Just let me know.

Jon

[Neil Kirkland]

The neighbors are correct, the links came up when I configured copper ports in the respective VLAN's on the switches and plugged something in - still can't reach vlan 2 from anywhere except vlan 5 (which is on the same switch) ...

I was kind of wondering because at first I configured as trunks - that failed, tried setting them as vlan spanning tree and that failed - because you can't have them as L2 and L3 at the same time .... so that failed ... never mind have all weekend to play.

Do I need to do something else to get ICMP / IGMP to run across the links (wouldn't have thought so but just asking).

Just a thought - don't I need some sort of ACL to permit VLAN 3, 4 and 5 to communicate with VLAN 2 and vica versa ?

[Jon Marshall]

Neil

No need for any extra config. You don't need acls and ICMP should just go across. 

You are pinging from the actual switch so SW1 should use it's 10.10.10.x address on the L3 link to ping the 10.10.2.253 address. So on SW1 when you do a "sh ip route" on SW1 can you see a route for the 10.10.2.0/24 subnet and is the next hop the 10.10.10.x address on SW3 ?

If so it should work. I don't know if the 10.10.2.253 address is an actual server or a test laptop but if it is a windows client make sure you turn off the host firewall as this would block the ping. .

If we concentrate on SW1 and SW3 it shouldn't take us long to work this out. Can you post a "sh ip route" from each switch and we can go from there.

I usually check in at the weekend anyway so i will be around but by all means post the outputs now and i'll have a look.

Jon

[Neil Kirkland]

Thanks - the grey matter's starting to melt here - this is sure good training for diagnosis ...

Ping works fine on SW3 from VLAN 5 but SW3 is where VLAN 2 is hosted (check out the drawing I posted).

It won't work from either 1 or 2 despite the neighbors showing up correctly ... just cleaning up and starting again - it's hard to keep track of what is and isn't set when you're jumping from switch to switch ... IP 253 is a VM running 2003 Server, 252 is an SQL instance on the laptop running the VM this is connected to VLAN 2, both laptop and VM have fixed IP address's. Have another laptop connected to SW3 on VLAN 3 and a PLC connected to VLAN 4 and 5 on SW 2 and 3 respectively

Fear not I'll post more - really appreciate your patience.

[Jon Marshall]

Neil

No problem. lots of patience left

Sounds like you are almost there. With a L3 setup like this the routing table is really where to look if things are not working.

So if there is route to the destination subnet with the correct next hop and there is a route on the other switch for the return traffic then it could well be something else.

Jon

[Neil Kirkland]

Well I had a moderate amount of success ... I got the trunks working, and figured out the loopbacks and SSH V2 for remote management, all functional, can ping and ssh all boxes and anything attached.

So moved on to DNS ... thats' a can of worms and no mistake - I'd have expected Cisco's to handle it better than they do - as in they don't.

I really wanted the dhcp servers to be 'local' but trying that fails miserably because none of the hosts 'register' with DNS properly and the DHCP won't do it for them ... nothings easy eh.

But I suppose I should start a different topic for that.

As to the problem - despite using a write erase to clean up the config and start again it turned out that the vlan's created previously when I was playing with spanning tree were still there - go figure - I just assumed that the router was putting them there, wrong they weren't getting erased and although they had no IP they were still causing trouble. Sounds crazy but using a no vlan # got rid and hey presto pings etc started working.

The only real annoyance is that the switch doesn't present a Vlan that nothing is connected to - despite being told 'no shut' having to stick a live cable in when you have multiple vlans is simply crazy - and murder to test.

Really appreciate the input Jon or I'd have spent even more hours chasing my tail.

Next step - deciding how to handle DHCP / DNS / NTP I really really didn't want a 'common' DHCP box but Cisco's poor handling of DNS looks like it will leave me no choice ...

[Jon Marshall]

Neil

So did you end up with trunks and not L3 connections ?

As for the erasing of the configuration, that as you found out does not get rid of the vlan database. You need to delete the vlan.dat file in flash for that to happen.

DHCP/DNS i actually prefer a dedicated Windows server for this as it all integrates well.

If you do use a central server then for all the vlans that are not in the DHCP server vlan you will need to add under the L3 vlan interface on each switch

int vlan

ip helper-address

Jon

[Neil Kirkland]

I call them trunks but they aren't, they are as you said L3 connections.

I already tried the ip helper address thing but the windows DHCP box refuses to serve from any scope except its own - i.e 10.10.2.253 - it won't serve 10.10.3.0/24 etc etc - the ip helper address is the DHCP server address, I have verified that it is reachable, the scopes are alive and well but only devices in vlan 2 (10.10.2.0/24) get ip address's.

I also need to set my L3 connections up as NTP peers I think - not played with the NTP yet - or should this pass across without any 'mods' ..

PPS : I looked at the variance thing too but I think that could result in the overload of a link - i.e SW1 will only calculate the load on the connections attached to it so it could overload the connections on the downstream switch - true / false?

[Jon Marshall]

Been a long time since i did NTP, need to have a quick read up on that

DHCP. As long as you have used the ip helper-address under the L3 vlan interfaces (not the L3 routed links) then it should work providing the scope is active which you have verified.

You may need to do some DHCP debugging on the switch.

So the DHCP server switch has routes back to all the vlan subnets on the other switches ?

Jon

[Neil Kirkland]

Yes from the switch hosting the DHCP box I can ping all the VLAN IP address's i.e. 10.10.3.254, 10.10.4.254, 10.10.5.254 .... assuming something is connected to them of course.

I need to get some packet capture running - been a long day though

[Jon Marshall]

Neil

Just a quick check.

On the ports that clients connect to do you have "spanning-tree portfast" configured ?

Jon

[Neil Kirkland]

if it is then it sure doesn't show up in show interfaces, I can set no spanning-tree explicitly if it will help ....

There are no 'BPDU' errors - as in BPDU doesn't exist in the list

[Jon Marshall]

Yes you should try setting it but only on ports that connect to end devices eg. PCs, servers etc. Note also if the server NIC was running as a trunk then it would actually be "spanning-tree portfast trunk".

But for normal clients/servers ie. the port is only in one vlan then "spanning-tree portfast" is enough.

What this command does is to allow the port to begin forwarding frames immediately. What can happen is without portfast applied the port has to go through the STP states before it can begin forwarding by which time DHCP has timed out and so the PC gives up.

It does depend which type of STP you are running but it is worth trying as it rules out one more possible cause.

To be honest DHCP generally just works once the "ip helper-address" is configured and there is full network connectivity which you now have.

Jon