Re: routing between vlans not working right

steverizkalla · ‎02-14-2011

Hi everyone. We recently had something happen to our multi switch and multi vlan network and don't know what what could be happening. I don't know how detailed I should get here but in general, we have a number of vlans that at one time could all communicate with each other. One vlan has all the DNS and DHCP servers and all the other vlans use a helper to get to the dhcp server. We have two 4507 switches, a 3750 and a 2960 I believe. I've gone through each config and cleaned up all the programming as best I could. ie... vlan subnet masks and helper addresses. anyway, here are a couple issues I am unable to fix...

We have the following vlans

- vlan2 is a migrated lan with the original 192.168.50.0 ip subnet and a /23 mask

- vlan3 is a server vlan with ip subnet 172.20.2.0/23 where all devices are statically assigned

- vlan4 is a user vlan with ip subnet 172.20.4.0/22 where all device IP addresses are DHCP assigned

- vlan6 is a DEV vlan with ip subnet 172.20.10.0/23 where all device IP addresss are DHCP assigned

vlan2 has a storage array named "lion" where each of 10 storage nodes have their own static IP addresses. The system alsp runs its own Name Server where when pinged, a random IP address out of 50 is returned. Therefore when the Name Server is hit, everyone gets attached to a different processing node. For instance, our storage array named "lion" is pinged. Lion is a name server where one of 50 IP addresses are returned, so when we wish to access this storage, we all get different IP addresses to access it therefore we are automatically balanced on the array so we are not all hitting the same storage processing node. ANYWAY, when we ping lion from any subnet other then the one where it resides, the server vlan, we do not get replies. So we can do a "ping lion" from any device on the server vlan and everything works. Each time we ping we get a random IP Address returned. Now when we ping from any of the other vlans, we do not get replies. What would block this from the other vlans? I am at a loss. This worked until about 2 weeks ago and now I Cannot figure it out.

We also have an ASA but I didn't think that would have any bearing on this....

Anything I should be targeting or looking at? Everything looks fine!!

What other information would be pertinent here for you?

Thank you for reading!!

Steve

bmcginn · ‎02-14-2011

Hi Steve,

Looks like a few troubleshooting steps are the go:

From the looks of it, layer 1 seems to be working, ie you can talk to lion from the same subnet which to me shows that layer 1 is ok. On the switch that the storage array is on, I assume you can ping lion (or at least one of the IP addresses it has )from the same vlan?

ie switch#ping ip 192.168.50.X source 192.168.50.G (where G is the VLAN SVI's ip address and X is one of Lion's IP addresses)

Can you ping devices on the VLAN that aren't lion?

Once you have determined that you can ping the IP from the VLAN its in, can you ping it from one of the other VLANs?

ie switch#ping ip 192.168.50.X source 172.20.4.U (where U is the User VLAN SVI's ip address)

Can you ping devices on the VLAN that aren't lion or do not have any IP address that Lion could use? If you can ping other devices on Lion's VLAN from the other VLANs then the problem will lie with Lion probably. (I am assuming that the switch that you are pinging from is the one that's doing the routing between Lion's VLAN and the User VLAN.)

If it doesn't work, check Lion's default gateway to make sure the layer 3 information is ok.

That may help a bit, but what would be very handy would be a quick diagram showing the devices, their IP address and LAN segments.

Brad

steverizkalla · ‎02-15-2011

Let me fix something I mistyped earlier.

FIX

VLAN3 the server vlan (not vlan2) has a storage array named "lion" where each of 10 storage nodes have their own static IP addresses. The system alsp runs its own Name Server where when pinged, a random IP address out of 50 is returned.

So yes layer 1 is working fine. From any device on 172.20.2.0 we can ping "lion" and get a random IP address returned properly. Here is a pin sample from a server on vlan3... I ran the ping 3 times and got 3 different IP addresses returned. Great. This worked fine. Layer 1 working.

C:\Documents and Settings\Administrator.HBSC>ping -n 2 lion

Pinging lion.HBSC.local [172.20.2.180] with 32 bytes of data:

Reply from 172.20.2.180: bytes=32 time<1ms TTL=64
Reply from 172.20.2.180: bytes=32 time<1ms TTL=64

Ping statistics for 172.20.2.180:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Administrator.HBSC>ping -n 2 lion

Pinging lion.HBSC.local [172.20.2.194] with 32 bytes of data:

Reply from 172.20.2.194: bytes=32 time<1ms TTL=64
Reply from 172.20.2.194: bytes=32 time<1ms TTL=64

Ping statistics for 172.20.2.194:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Administrator.HBSC>ping -n 2 lion

Pinging lion.HBSC.local [172.20.2.190] with 32 bytes of data:

Reply from 172.20.2.190: bytes=32 time<1ms TTL=64
Reply from 172.20.2.190: bytes=32 time<1ms TTL=64

Ping statistics for 172.20.2.190:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Documents and Settings\Administrator.HBSC>

---------------------------------------

The SVI is essentially the VLAN gateway IP address? I am reading up on that now...

From vlan3 that lion is on, we can ping from lion or any other device to any other device on any other vlan successfully. It is only the virtual IPs on lion that don't appear to be pingable from other vlans. I will go through all of lion's configs to be sure they are fine. maybe the problem does lie within lion...

---------------------------------------

I have another issue with VLAN4, the user VLAN where the users stopped obtaining IP addresses from the DHCP server. I am starting to look into that now. That same DHCP server gives out IP addresses to all the other VLANs just fine so that will be another struggle.

Thank you for the tips. I will report back what I find....

Steve

steverizkalla · ‎02-15-2011

In trying to diagnose some VLAN4 issues where now when we put a computer on VLAN4, the computer's CPU gets pegged to 100% on process "system". So this is a reason I have moved everyone off VLAN4 and onto another one that doesn't hang anyone up. Ugh...

I see these lines in the switches. They have been configured to uplink an aggregate connection to an end switch. Here are the interface configs. My questions is since these ports are only moving VLAN info to the end switch, does the VLAN4 issuance even need to be in these port configs? Does it hurt anything?

SW 1 (3560) ports 46 and 48 uplink config to SW 2 (2960)

interface GigabitEthernet0/46
switchport access vlan 4
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode desirable
spanning-tree portfast
!
interface GigabitEthernet0/48
switchport access vlan 4
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode desirable
spanning-tree portfast

SW 2 (2960) ports 45 and 46 downlink config from SW1 (3560)

interface GigabitEthernet0/45
switchport mode trunk
channel-group 2 mode desirable
!
interface GigabitEthernet0/46
switchport mode trunk
channel-group 2 mode desirable

glen.grant · ‎02-15-2011

Having that vlan 4 statement in their doesn't hurt anything , just says if the link was not a trunk it would be an access port in vlan 4 , but it is hardcoded as a trunk so it should not hurt anything. You should not have spanning tree portfast on ports that connect 2 separate switches though , remove that .

steverizkalla · ‎02-16-2011

Thank you Glen. I actually may close this post as I believe we have a bigger problem that actually may solve all the other issues I've been experiencing.

Core Switch: 4507R running v12.2 with (2) WS-X4013+10GE Supervisory Engines with a couple 10GB links to other switches, (5) 48 port switch cards

- At first it was devices on VLAN4 (User) that stopped obtaining an IP address from the DHCP server and having random outages across VLANs for some minutes then access is restored like nothing happened

- Then nothing on VLAN4 was working at all so I moved everyone to the VLAN6 (QA/DEV) just to get them up and running to be able to get email, print and use the web; printers that were on VLAN4 had to be moved to VLAN6 as they stopped working too

- Then users on VLAN6 (QA/DEV) started having outages accessing some servers and an ERP system on VLAN3 (ORIG Net) where there are some older servers

- So I then moved some users that required access to the ERP over to VLAN3 becuase outages between VLAN6 and VLAN3 started cropping up

- Now the network isn't being used as it was intended

I am now thinking since the programming appears fine maybe there is some hardware issue that I need to diagnose. I can post the config if anyone would like to see...

The switch has been pretty static and these things started just happening so I am targeting hardware.... I have a second 4507R that I can probably rob parts from since we've downsized considerably...

thoughts?

Thanks!

Steve

steverizkalla · ‎02-16-2011

If I can add a couple other random items we've seen....

- we can login to the Cisco VPN configured on the ASA. Sometimes we can see internal resources, other times we login and can't ping or see anything but we can in fact login; the "inside" interface on the ASA is plugged into port 3/1 on the core 4507R

- or is it possible there is some device on the network somewhere simply going crazy packetwise...? I don't think so but I've seen stranger things

- I have Orion By Solarwindows loaded on one server with all the pertinent interfaces being monitored but I am not sure what I'm looking at. I am not trained in this. I added the interfaces and am now trying to figure out what I'm looking at... I hope this shows something!

thanks!