3560CG failing to open ports

[Matthew Lucas]

I'm becoming a real pest on here, haha.

I have a number of 3560CG-8PC-S switches. My intention for them is to act as kind of gateway L3 switches - one for each satellite site. My thinking was simply to have an L3 device at the gateway to each of those sites so that any inter-vlan traffic within each site can stay within the site rather than having to traverse the relatively slow radio links to get back to the 3750X stack in the core. They are also, however, going to be directly serving client devices

My issue is that for some reason, when connecting a new device (laptop etc) to one of the access ports on the 3560's, the port behaves as if it's being blocked. No DHCP addresses go through, the indicator remains orange, and the clients have no connectivity. However, if I wipe the config, I get a VLAN 1 IP address for my client no problems at all. And to make matters more confusing, only two out of my four 3560's are doing this. The other two have exactly the same config, but work perfectly.

To that end, I'm loading the config below. I've followed that by the show running-config output, and show ip interface brief outputs.

configure terminal

hostname ASW34

!

enable secret *RuT1l3&

service password-encryption

username xxxx password xxxx

!

ip domain-name sierra-rutile.local

crypto key generate rsa

1024

line vty 0 1

login local

transport input ssh

!

line vty 2 15

transport input none

exit

!

!

line con 0

login local

exit

no ip http server

no ip http secure-server

!

vtp mode client

vtp domain sierra

vtp password rutile

!

ip default-gateway 10.0.4.10

!

ntp server 10.0.4.10

!

interface vlan 1

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

no shutdown

exit

!

interface vlan 4

ip address 10.0.4.41 255.255.252.0

exit

!

interface vlan 8

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 16

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 20

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 24

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 28

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 32

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 36

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 248

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

interface vlan 252

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

exit

!

ip routing

ip route 0.0.0.0 0.0.0.0 10.0.4.10

!

access-list 101 deny ip 192.168.8.0 0.0.0.255 10.0.0.0 0.255.255.255

access-list 101 permit ip 192.168.8.0 0.0.0.255 any

interface vlan 244

ip access-group 101 in

!

interface range g0/9 -10

switchport trunk encapsulation dot1q

switchport mode trunk

exit

!

interface range g0/1 -8

switchport mode access

switchport access vlan 12

spanning-tree portfast

exit

!

spanning-tree mode rapid-pvst

end

write memory

Now a running config copy from the NOT working switch. This is identical (i've gone through them both line by line side by side) with one of the other identical switches that IS working.

SW31#sh run

Building configuration...

Current configuration : 3386 bytes

!

! Last configuration change at 04:31:31 UTC Fri Apr 1 2011

! NVRAM config last updated at 04:31:31 UTC Fri Apr 1 2011

!

version 15.0

service config

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname ASW31

!

boot-start-marker

boot-end-marker

!

enable secret 4 5fpDlu4LdCozFYxrLimWlqRSZLorgqR1LnuU34XhHaE

!

username xxxx password 7 11434A2B1043055F57186D

no aaa new-model

system mtu routing 1500

ip routing

!

!

ip domain-name sierra-rutile.local

!

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

!

interface GigabitEthernet0/1

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/2

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/3

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/4

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/5

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/6

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/7

switchport access vlan 12

switchport mode access

spanning-tree portfast

!        

interface GigabitEthernet0/8

switchport access vlan 12

switchport mode access

spanning-tree portfast

!

interface GigabitEthernet0/9

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface GigabitEthernet0/10

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface Vlan1

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan4

ip address 10.0.4.41 255.255.252.0

!

interface Vlan8

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan16

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan20

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan24

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan28

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130!        

!

interface Vlan32

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan36

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan244

no ip address

ip access-group 101 in

!

interface Vlan248

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

interface Vlan252

no ip address

ip helper-address 10.0.4.129

ip helper-address 10.0.4.130

!

ip default-gateway 10.0.4.10

no ip http server

no ip http secure-server

!

ip route 0.0.0.0 0.0.0.0 10.0.4.10

!

access-list 101 deny   ip 192.168.8.0 0.0.0.255 10.0.0.0 0.255.255.255

access-list 101 permit ip 192.168.8.0 0.0.0.255 any

!

!

!

line con 0

login local

line vty 0 1

login local

transport input ssh

line vty 2 4

login

transport input none

line vty 5 15

login

transport input none

!

ntp server 10.0.4.10

end

And finally, show ip interface brief:

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  unassigned      YES manual up                    up     

Vlan4                  10.0.4.41       YES manual up                    up     

Vlan8                  unassigned      YES unset  up                    up     

Vlan16                 unassigned      YES unset  up                    up     

Vlan20                 unassigned      YES unset  up                    up     

Vlan24                 unassigned      YES unset  up                    up     

Vlan28                 unassigned      YES unset  up                    up     

Vlan32                 unassigned      YES unset  up                    up     

Vlan36                 unassigned      YES unset  up                    up     

Vlan244                unassigned      YES unset  up                    up     

Vlan248                unassigned      YES unset  up                    up     

Vlan252                unassigned      YES unset  up                    up     

GigabitEthernet0/1     unassigned      YES unset  down                  down   

GigabitEthernet0/2     unassigned      YES unset  down                  down   

GigabitEthernet0/3     unassigned      YES unset  up                    up     

GigabitEthernet0/4     unassigned      YES unset  down                  down   

GigabitEthernet0/5     unassigned      YES unset  down                  down   

GigabitEthernet0/6     unassigned      YES unset  down                  down   

GigabitEthernet0/7     unassigned      YES unset  down                  down   

GigabitEthernet0/8     unassigned      YES unset  down                  down   

GigabitEthernet0/9     unassigned      YES unset  up                    up     

GigabitEthernet0/10    unassigned      YES unset  up                    up     

[visitor68]

where's the routed vlan interface for vlan 12?

[Matthew Lucas]

There is no VLAN 12. I've left the space to allow for growth in VLAN 8 - we're at about 75% address space usage as things stand, and with the increase in mobile devices, it wouldn't surprise me to hit the barrier, so I've left a gap. And it seemed easier to keep VLAN names in accordance with the third octet (or vice versa) so 10.0.16.0 is VLAN 16.

[Bilal Nawaz]

Hi Matthew, If i recall correctly about the DHCP problem you was having before mentioned in that post, you do not need the ip helper address on your access switches - you do not need the vlan interfaces either (you only need vlan's and trunks on your access switch). The vlan interface and helper addresses should be on your core which is your 3750X's i think.

Have you tried plugging in one of the devices that is working in to the switch that isn't working. Just as an observation I also noted that you are on IOS v15 on the not working switch... and what about the other one that's working?

Also what is the status of the port, e.g. show interface gix/y

Could you also do a 'show vlan' please just to double make sure the vlans got created.

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[pille1234]

Hi,

I have a number of 3560CG-8PC-S switches. My intention for them is to act as kind of gateway L3 switches - one for each satellite site. 

What do you mean with "kind of gateway"? The switch either is a gateway or it is not. As it is configured right now it is not, because your L3 interfaces (SVIs) don't have IP adresses assigned. As a result the ip helper config has no effect. What IP should the switch use as giaddress for the relayed packet? If you want to have a L3 termination for each sattelite, which is usually a good design, than you must configure an IP address for the SVIs and of course your IP subnet design needs to reflect that, meaning you need a single subnet for each vlan for each sattelite.

[Matthew Lucas]

Haha, you're exposing the level of knowledge I don't have, pille1234.

Ok, the intention is to have them as an actual L3 gateway. So, if I'm understanding you correctly, the gateway L3 device needs to have an IP address within the correct range for each SVI - which is fine - I have that available. Currently each satellite has its own subnet, corresponding to a dedicated VLAN. VLAN 4 is the mgmt VLAN and is across all sites, as is 248 (guest wifi).

So, taking your suggestion along with Bilal's - I need to strip out the SVI's (and ip helper-addresses) on the L2 access switches because they're unnecessary, and ensure each SVI on the L3's has its own IP.

At the risk of exposing my ignorance even further ... the "giaddress"? Not familiar with the term.

Site IP schema is:

VLAN 1 (keeping it on temporarily for legacy devices which are being phased out in a month or two) 10.0.0.0/24

VLAN 4 (mgmt) 10.0.4.0/22

VLAN 8 (main site - core switch is here) 10.0.8.0/22

VLAN 12 space reserved for expansion by VLAN 8

VLAN 16 (satellite 1) 10.0.16.0/22 - two radio links to main site due to topography, thus two L3 devices

VLAN 20 (satellite 2) 10.0.20.0/22

VLAN 24 (satellite 3) 10.0.24.0/22 - very small satellite site - 12 users thus no L3 gateway. I know the scope is over the top, but keeps it easy to remember

VLAN 28 (satellite 4) 10.0.28.0/22 - future site, so space reserved, but no eqpt

VLAN 32 (satellite 5) 10.0.32.0/22 - future site, so space reserved, but no eqpt

VLAN 36 (satellite 6) 10.0.36.0/22 - L3 gateway

VLAN 244 192.168.0.0/24 (CCTV) currently at main site only, no DHCP req'd

VLAN 248 192.168.4.0/23 (Guest wifi) - across all sites

VLAN 252 10.0.252.0/22 (VoIP) - to be configured later

All advice gratefully received - I'll add credits to you all on my MOTD

[Matthew Lucas]

Presumably then any further L2 switches on the satellite sites need to have their local satellite L3 device as the default gateway, rather than the 3750X stack in the core? Which means I need to adjust the DHCP scopes for the satellite sites to indicate the correct default gateway?

How does that work then for multi-site VLANs?

[Bilal Nawaz]

So I think there are a few things to be dealt with here.

• Your link between sites... layer 3...

• Your access switches with ports not coming 'up'.

Between sites - Site A and Site B lets say we have Router_A and Router_B respectively.

Between Router_A and Router_B you can do routing - lets say ospf as an example...

Router_A would have an adjacency with Router_B and they both would have routes to get to each other from all local networks.

You would have to have different vlans/ip address schemes for the sites. Your helper addresses can point towards the same server. (as long as you have a route to get to it) YES! you will need to change default router address on DHCP for the other site.

If you need me to clear this further please do ask....

With regards to your switches - are the vlans created locally on the switch? What is the status of the port? What is the show interface fa0/1 (access port output)

Is there a layer 3 link that is 'up' between sites. Are you able to ping from site to site?

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[Bilal Nawaz]

So these IP's that are on your 3750X could instead be locally to the satellite sites...

Router_B will have 10.0.16.10/22

Router_C will have 10.0.20.10/22

Router_D will have 10.0.24.10/22

Router_E will have 10.0.28.10/22

and so on....

Of course this means you need router's / layer 3 capable devices per site in addition to access switches?

I think a bit more understanding is required of the topology between sites, like which site connects to which?

Kind of like this.... But ofcourse you dont have to use 2950's or any models specified below, Just an example.

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[Matthew Lucas]

Right, so let me get this straight.

My layer 3 switches do have adjacency. I'm not using a routing protocol because there's no choice of routes - each gateway has only a single route back to the core, so as long as I configure a default route of 0.0.0.0 0.0.0.0 10.0.4.10 (being the core switch) for each of the satellite layer 3 switches, provide an ip address and ip helper-addresses

for each of the SVI's and finally a default gateway on the switches themselves pointing to the core switch then that covers the layer 3 side of life, correct?

All satellite layer 2 switches simply need a default gateway of their local satellite layer 3 switch, and no SVI's except for VLAN 4 (management).

All satellite-site clients need their DHCP scopes to reflect that their default gateway must be their local satellite gateway switch.

Still not clear how I apply that on multi-site VLANs, like guest wifi? Or do I just make a new guest wifi VLAN for each satellite site, with the appropriate default gateway?

I'm in a lab environment at the minute, but all trunk links work and I have full connectivity (ping and DHCP) between sites/VLANs, with the exception of those two 3560s that aren't playing nice.

VLANs are not created locally, no - they're propagated via VTP from the core switch. That works fine over the trunk links.

[Bilal Nawaz]

With your multi-site VLANs I'm not sure what the best practice is... Since the very nature of guest vlan's CCTV too you could leave them as it is (they will exist everywhere i assume and be fairly static)

or just take the same approach in creating new vlans....

But if it's only layer 3 between sites then the only option is to have local vlans... Why? because your interconnects will be layer 3 which is routed - not layer 2 (trunking vlans) So there will be no mechanism to carry your vlans to the other site.

Your default routes will do fine.

You pretty much got it in your summary above. Apart from - you will need the SVI for vlan 4 / WAN interface AND the local vlan(s).

When it comes to creating vlans per site, VTP will not work since layer 3 will be interconnecting your sites, not layer 2.

Can you verify which IOS you are on the working and non working 3560's. I assume you managed to get at least one port 'up' as the trunk.

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[Matthew Lucas]

Ok, so a bit of thought needed as to whether I sacrifice the concept of layer 3 between satellite and main sites in the interests of simplicity, keep my VLANs central and propagated via vtp, in which case my 3560's become access switches pure and simple not running any layer 3 at all. VTP pruning should keep unnecessary traffic to a minimum anyway, shouldn't it?

Or do I keep the gateway switches on layer 3, remove the unnecessary vlans from the core switch (all satellite vlans but retain mgmt and cctv), leave my default route as is and create the satellite vlans on the satellite gateway switches and propagate from there on vtp?

I'm slightly concerned that if I do layer 3 between sites, my new enterprise wireless mesh kit (aerohive) is going to kick off because it doesn't have the management server on the same network... in fact that's the case in the current design anyway. Damn! haha. Need to get in touch with Aerohive support now, just to make sure.

All 3560's have been updated to the latest IOS - 15.0.2 (SE2) - both working and non-working. There's something odd going on there - I don't believe it's a hardware fault. Far more likely that I've overlooked something. I'll try pulling the config off the working ones via TFTP and drop them on the non-working ones.

Guys, thanks so much for your input. Really elevated my understanding today. I'd buy you all a beer/poison of your choice, but geography's rather in the way, so I'll settle for a big thanks.

[Bilal Nawaz]

If it's simplicity that you are after, I'd say just stick with the layer 2 concept. Save's you from much headache. But not sure if this is best practice to follow.

You dont have the problem of spanning-tree since all will be 'forwarding and or root ports' which is normally the biggest concern when spanning vlans across sites.

The switch problem is very strange. Since you have spanning-tree portfast enabled, the port should go forwarding straight away. Do you see anything in the logs when you plug a device into one of them ports?

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[Matthew Lucas]

I think my feeling is to go for simplicity. It's not the most elegant solution, but I have to take into account that these guys will have to manage it and expand it later on, and they've only just about wrapped their heads around the concept of VLANS and trunk and access ports. Throwing in a load of layer 3 stuff is going to complicate matters, and since all core infrastructure (servers etc) is all exactly that - at the core, then essentially I can get away with having a rather large (geographically-speaking) LAN, with some slow links here and there. Once I make it a WAN and expect them to manage it, they're going to be in a world of hurt.

There aren't that many users/devices on the satellite sites - probably a few hundred at best, and the volumes of traffic involved aren't vast - it's only email, web browsing and a bit of file sharing. It's a very basic setup, so going for needlessly complex purely in search of elegance is unnecessary at this stage. Come 5 years down the line, it may be a different story, but there's only so far you can plan ahead.

In the interests of closing this thread properly, I'll let you know where I get with the 3560's - though it's not going to be for a week now as I have to go offsite. Thanks all again for your help. Much appreciated.

[pille1234]

My layer 3 switches do have adjacency. I'm not using a routing protocol because there's no choice of routes - each gateway has only a single route back to the core, so as long as I configure a default route of 0.0.0.0 0.0.0.0 10.0.4.10 (being the core switch) for each of the satellite layer 3 switches, provide an ip address and ip helper-addresses
 for each of the SVI's and finally a default gateway on the switches themselves pointing to the core switch then that covers the layer 3 side of life, correct?

correct

All satellite layer 2 switches simply need a default gateway of their local satellite layer 3 switch, and no SVI's except for VLAN 4 (management).

That depends. Is your management-vlan supposed to be spanned across all satellites like the guest wifi? In that case your Core switch is the central gateway for all switches (management-wise). If you use a separate management vlan for each satellite than it is configured the same way as the user vlans with the L3 gateway beeing the satellite boundary switch.

Still not clear how I apply that on multi-site VLANs, like guest wifi? Or do I just make a new guest wifi VLAN for each satellite site, with the appropriate default gateway?

Same as above: if you decide to use one big cross satellite vlan (wich I suggest for WLAN with mobile devices)  than the core switch is the only switch with an L3 interface for that vlan. All other switches are plain L2 switches in regard to this specific vlan (= no SVI).