Re: A Weird Inter-vlan Routing Problem

lamav · ‎07-24-2008

Folks:

I have a 6509 switch running 12.2(18)SXF8 with about 5 vlans configured on it in L2 and L3 (SVI).

Hosts on different vlans cannot PING each other successfully. So, a host on vlan 120 cannot ping a host on vlan 124. In fact, I cant even do a successful extended PING to a device on any given vlan if I source it from the SVI of another vlan on the switch itself.

What I CAN Do is run a successful PING to any given SVI and source another SVI on that switch.

I have cleared the ARP and MAC tables already. No dice.

Any ideas off the top of anyone's head?

Thanks

VL

Edison Ortiz · ‎07-24-2008

Let's see the show ip route and show ip interface brief | ex una from the switch along with an ipconfig /all output from workstation(s) on different subnet.

__

Edison.

lamav · ‎07-24-2008

Thanks...

6509>sh ip ro

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

192.168.127.0/30 is subnetted, 3 subnets

C 192.168.127.132 is directly connected, Vlan3

C 192.168.127.128 is directly connected, Vlan12

D 192.168.127.136 [90/3072] via 192.168.127.130, 12:44:07, Vlan12

C 192.168.126.0/24 is directly connected, Vlan126

192.168.250.0/29 is subnetted, 1 subnets

C 192.168.250.0 is directly connected, Vlan250

10.0.0.0/24 is subnetted, 1 subnets

D EX 10.1.1.0 [170/130816] via 192.168.127.130, 12:19:41, Vlan12

192.168.3.0/26 is subnetted, 1 subnets

S 192.168.3.0 [1/0] via 192.168.250.4

C 192.168.122.0/23 is directly connected, Vlan122

C 192.168.120.0/23 is directly connected, Vlan120

S 192.168.120.0/21 is directly connected, Null0

S 192.168.160.0/20 [1/0] via 192.168.250.4

C 192.168.124.0/23 is directly connected, Vlan124

R 192.168.0.0/23 [120/1] via 192.168.127.134, 00:00:04, Vlan3

6509>

CNFsgMUSalpbu01>sh ip int brief

Interface IP-Address OK? Method Status Protocol

Vlan1 unassigned YES NVRAM administratively down down

Vlan3 192.168.127.133 YES manual up up

Vlan12 192.168.127.129 YES manual up up

Vlan120 192.168.120.2 YES manual up up

Vlan122 192.168.122.2 YES manual up up

Vlan124 192.168.124.2 YES manual up up

Vlan126 192.168.126.2 YES manual up up

Vlan250 192.168.250.2 YES manual up up

GigabitEthernet1/1 unassigned YES unset down down

GigabitEthernet1/2 unassigned YES unset down down

GigabitEthernet1/3 unassigned YES unset down down

GigabitEthernet1/4 unassigned YES unset down down

GigabitEthernet1/5 unassigned YES unset up up

GigabitEthernet1/6 unassigned YES unset down down

GigabitEthernet1/7 unassigned YES unset down down

GigabitEthernet1/8 unassigned YES unset down down

GigabitEthernet1/9 unassigned YES unset down down

GigabitEthernet1/10 unassigned YES unset down down

GigabitEthernet1/11 unassigned YES unset down down

GigabitEthernet1/12 unassigned YES unset down down

6509>

Still need to get into the servers. I can say that they are having a problem with many servers on several different vlans.

Also, they do have a FWSM installed, which I havent been able to access either. SO I am wondering if proxy arp has to be disabled on the FW because [maybe] the FW is responding to ARP requests and the servers are forwarding to the FW instead of the switch SVI. (Just a suspicion...cant confirm until I can get into it.) I would like to see the arp table for the servers, too.

Thanks

Victor

Edison Ortiz · ‎07-24-2008

Also, they do have a FWSM installed,

This can be the cause. I'm not a FWSM expert so I'll to bow out of this one if you need help on that :)

lamav · ‎07-24-2008

Thanks.

I still need to get into the FWSM and the servers themselves.

meanwhile, has anyone else encountered this problem?

I remember troubleshooting this exact same kind of problem with Jon Marshall for someone who posted the exact same problem. The thread went on for a week....dont know what the final outcome was.

Jon, do you remember?

VL

Edison Ortiz · ‎07-24-2008

This is the thread?

http://forum.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=LAN%2C%20Switching%20and%20Routing&topicID=.ee71a04&fromOutline=&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.2cbef1c1

lamav · ‎07-24-2008

No, but this one may be even more pertinent because there is an FWSM involved.

That other thread involved a female poster who was not using one. It occurred in April or so...around that time, I think. And there were about 40 messages on the thread.

This is great, though.

Let me check it out...

Thanks, Edison...

Victor

Edison Ortiz · ‎07-24-2008

Jon has worked on many FWSM threads. Use the "Search NetPro" tool on the upper right and enter "Jon FSWM". You should be able to find other threads on this subject.

HTH,

__

Edison.

lamav · ‎07-24-2008

Thats what I am saying..It was NOT an FWSM problem.

It was an inter-vlan routing problem that manifested itself exactly the way mine is...to the best of my recollection.

There was no FWSM involved.

Thanks

Victor

lamav · ‎07-24-2008

I got into the FWSM.

It looks clean.

All traffic between vlans 120-126 are routed locally in the switch.

All traffic destined for vlans 160 and above are forwarded to the FWSM. They sit (logically speaking) on the "outside" -- untrusted network.

So, it's not a routing issue involving the FWSM. All traffic sourced from vlan 120 and destined for vlan 124 (and vise versa) should be routed locally.

Anyone have any ideas with the added info Ive given?

The problem is no inter-vlan routing and the FWSM doesnt seem to be the problem.

I can route between SVIs on the different vlans, though.

Thanks

Victor

Edison Ortiz · ‎07-24-2008

Are the workstations connected directly to this 6509 switch?

You can verify the Vlans aren't being pruned downstream to the access switches.

Create two access ports in the switch itself between the trouble vlans and connect a workstation on each port, see if it works there.

__

Edison.

lamav · ‎07-24-2008

Good thought, Edison.

The workstations actually plug directly into this bad boy...

They're trying a code upgrade tonight...lets see what happens.

Thanks

Victor

lamav · ‎07-25-2008

Edison, check it out...

The reason why the servers on different vlans/subnets could not communicate with each other was that they did not have routes built in their routing tables to go off-net.

That was my initial diagnosis and I was pretty certain of it. But the client kept telling me that they have this issue with every server on the different LANs ( a couple dozen). At that point I thought perhaps there was indeed a different problem.

As it turns out, the servers were all designed to communicate only with other servers on their own subnet.

Sigh...

So, the switch was fine, the servers were hosed.

Thanks

Victor

Edison Ortiz · ‎07-25-2008

Victor,

Thanks for the post back.

__

Edison.