3750 Routing Issue

JohnTylerPearce · ‎12-01-2011

I have a topology as configured below

3750StackA -----> 3750StackB(Core Switch/Root Switch) -----> 3750StackC(Data Center)

The old way this was configured, is that every 3750 switch A,B, and C are running at L3. They have SVIs for all vlans in our enterprise network.

Considering the amount of ICMP redirects and inefficent routing that I witnessed I decided to have "B" as the Core Switch for all networks, and

house the only L3 SVI for the switches. As of this writing all networks have a x.x.x.1 default gateway on that switch. I've been slowly killing

the SVIs on the other switches to make A and C complete L2 devices. I did this with a vlan we will call VLAN 210 which worked perfectly.

I killed the two vlan interfaces for vlan 210 off of A and C and communication internally and externally work perfectly. But for the rest of are

networks, I can kill the ip address off of the SVI on A, and everything works perfectly internally and externally, but as soon as I kill the ip

address on the SVI of C external communication drops but inside is completely fine. We have an ASA that holds the ISP connection, and

I don't see anything pointing to the old SVIs on A and or C, I checked this before even attempting this.

The vlans are configured to go across the trunks between switches and B has a route to everything in the network. I tried to ping 8.8.8.8

when I killed the ip address on one of the SVIs on A, and I noticed on the capture I setup on my ASA, that it sees the IP address of

VLAN 210 and it goes out to 8.8.8.8 (echo request), I also see (echo replies) coming back from the ASA which is connected to C.

I succesfully ping devices on B, but nothing on A. I've tried looking at everything I can, and it's driving me nuts. I don't see anything in

the cef, mac, or arp tables that would really lead me on to anything.

Anyone have any ideas?

I personally think the problem is between A and B I just can't quite figure out what's causing it.

Ven Taylor · ‎12-01-2011

Just to clarify. Your ASA (and Internet connection) is connected to switch C.

It actually sounds like the problem is L3 routing between the ASA and switch C's SVI's.

Can you post those configs?

Ven

Ven Taylor

JohnTylerPearce · ‎12-01-2011

Yes, Ven, the ASA (Internet Connection) is connected to switch C, sorry if I didn't clarify that. I'll post the configs tomorrow, since I'm headed home for the day. Thanks for looking in to it Ven.

JohnTylerPearce · ‎12-02-2011

ASA Route Table
---------------

route outside 0.0.0.0 0.0.0.0 x.110.215.1 1
route inside 10.10.10.0 255.255.255.224 x.110.215.17 1
route inside 172.16.64.0 255.255.240.0 x.110.215.17 1
route inside 192.168.5.0 255.255.255.0 x.110.215.17 1
route inside 192.168.15.0 255.255.255.0 x.110.215.17 1
route inside 192.168.20.0 255.255.255.0 x.110.215.17 1
route inside 192.168.21.0 255.255.255.0 x.110.215.17 1
route inside 192.168.60.64 255.255.255.224 x.110.215.17 1
route outside 192.168.253.0 255.255.255.0 x.110.215.1 1
route inside x.110.208.0 255.255.255.0 x.110.215.17 1
route inside x.110.208.5 255.255.255.255 x.110.215.19 1
route inside x.110.209.0 255.255.255.0 x.110.215.17 1
route inside x.110.210.0 255.255.255.0 x.110.215.17 1
route inside x.110.211.0 255.255.255.0 x.110.215.17 1
route inside x.110.212.0 255.255.255.0 x.110.215.17 1
route inside x.110.213.0 255.255.255.0 x.110.215.17 1
route inside x.110.214.0 255.255.255.0 x.110.215.17 1
route inside x.110.216.0 255.255.255.0 x.110.215.17 1
route inside x.110.217.96 255.255.255.224 x.110.215.17 1

interface GigabitEthernet0/0
description xxxxxxxxxx
speed 1000
duplex full
nameif outside
security-level 0
ip address x.110.215.4 255.255.255.240 standby x.110.215.5
!
interface GigabitEthernet0/1
description xxxxxxxxxx
speed 1000
duplex full
nameif inside
security-level 100
ip address x.110.215.17 255.255.255.240 standby x.110.215.18
!
interface GigabitEthernet0/2
description LAN Failover Interface
speed 1000
duplex full
!
interface GigabitEthernet0/3
description STATE Failover Interface
speed 1000
duplex full
!
interface Management0/0
speed 100
duplex full
nameif management
security-level 100
ip address 192.168.1.49 255.255.255.0 standby 192.168.1.62

Switch C Route Table
----------------
Gateway of last resort is x.110.215.17 to network 0.0.0.0

S    192.168.12.0/24 [1/0] via x.110.215.45
S    192.168.13.0/24 [1/0] via x.110.215.45
C    192.168.15.0/24 is directly connected, Vlan915
S    192.168.9.0/24 [1/0] via x.110.215.45
S    192.168.10.0/24 [1/0] via x.110.215.45
S    192.168.40.0/24 [1/0] via x.110.211.1
     172.16.0.0/22 is subnetted, 1 subnets
C       172.16.64.0 is directly connected, Vlan6
S    192.168.55.0/24 [1/0] via x.110.215.45
S    192.168.21.0/24 [1/0] via x.110.211.11
S    192.168.20.0/24 [1/0] via x.110.215.45
     x.110.212.0/24 is variably subnetted, 2 subnets, 2 masks
S       x.110.212.123/32 [1/0] via x.110.212.3
C       x.110.212.0/24 is directly connected, Vlan5
     192.168.5.0/24 is variably subnetted, 10 subnets, 2 masks
S       192.168.5.64/32 [1/0] via x.110.211.11
S       192.168.5.45/32 [1/0] via x.110.211.11
S       192.168.5.46/32 [1/0] via x.110.211.11
S       192.168.5.40/32 [1/0] via x.110.211.11
S       192.168.5.35/32 [1/0] via x.110.211.11
S       192.168.5.63/32 [1/0] via x.110.211.11
S       192.168.5.0/24 [1/0] via x.110.211.11
S       192.168.5.22/32 [1/0] via x.110.211.11
S       192.168.5.17/32 [1/0] via x.110.211.11
S       192.168.5.19/32 [1/0] via x.110.211.11
     x.110.215.0/28 is subnetted, 2 subnets
C       x.110.215.32 is directly connected, Vlan218
C       x.110.215.16 is directly connected, Vlan215
S    192.168.7.0/24 [1/0] via x.110.215.45
C    x.110.209.0/24 is directly connected, Vlan209
C    192.168.50.0/24 is directly connected, Vlan950
C    x.110.208.0/24 is directly connected, Vlan2
C    192.168.1.0/24 is directly connected, Vlan1
C    x.110.211.0/24 is directly connected, Vlan211
     x.0.0.0/27 is subnetted, 1 subnets
S       x.254.111.32 [1/0] via x.110.211.11
S*   0.0.0.0/0 [1/0] via x.110.215.17

Vlan1 192.168.1.34 YES TFTP up up

Vlan2 x.110.208.17 YES TFTP up up

Vlan5 x.110.212.4 YES TFTP up up

Vlan6 172.16.64.5 YES TFTP up up

Vlan209 x.110.209.5 YES manual up up

Vlan211 x.110.211.5 YES TFTP up up

Vlan215 x.110.215.19 YES TFTP up up

Vlan216 x.110.216.9 YES TFTP down down

Vlan218 x.110.215.33 YES TFTP up up

Vlan300 192.168.3.105 YES TFTP down down

Vlan915 192.168.15.4 YES TFTP up up

Vlan950 192.168.50.2 YES TFTP up up

Ven Taylor · ‎12-02-2011

Can you post the SwitchC config with SVI's?

Two more questions...

Can you move the ASA connection to SwitchB?

You really should have the core be the closest switch to the ASA.

If you can't do that, make sure you're trunking all your vlans between B & C.

Can you ping the ASA from switch B when you turn down the SVI's on C?

Ven

Ven Taylor

JohnTylerPearce · ‎12-02-2011

I updated the configuration for you. I understand completely what you mean about moving the core as close as possible to the ASA, but I can not do that. The whole network setup was done not by me, but several network engineers, and most of it by my predecessor.

I can ping the ASA from B when I kill the SVI IP on C. Although, I'm assuming it's doing this with source ip address of my management vlan with is vlan 1 (not my choice, that will be changed later).

I just killed the IP from a SVI on A, and from a host connected to A, I can ping B just fine, and I can also ping a server which is connected to C. But I cannot ping the inside interface on the ASA, which is the root of the problem.

Once again, Ven, thanks for taking a look at it.

JohnTylerPearce · ‎12-02-2011

From a host connected on A pinging the communication should take the following path.

1. The destination to x.110.215.17 is not local, so it needs to sent it to it's default gateway, It will arp for it's default gateway

which is hosted on B. A will then send the frame to B.

2. Frame will decapsulate the frame, look at the IP header and come to the conclusion that it has a directly connected

network for x.110.215.17, so it will send an ARP broadcast down all ports that belong to that specific VLAN.

3. The vlan is allows on th trunks between all three switches (trust me if it wasn't I would know about it, Plus I have verified it.)

4. The port on C which is connected to the inside interface on the ASA, should receive this ARP, and then the ASA should

have to send the echo reply to it's default gateway.

There must be some issue between the ASA and C. So I agree with you on that Ven. I just can't figure out how killing the

SVI IP address on A for this vlan for instance will kill that communication.

Ven Taylor · ‎12-02-2011

No problem. Can you look at your SVI configs as well as your SwitchC - Switch B interface config and your SwitchC - ASA interface config? Make sure your L2 trunking is good.

I see a static route on switch C that points directly to the ASA. Is that route being propagated to switch A or B?

Do you have redistribute static on switch C?

Check your routes on switch B and make sure nothing is pointing directly to switch C.

Ven

Ven Taylor

JohnTylerPearce · ‎12-02-2011

The entire routing here is done via static. We do not have any dynamic routing protocols.

A,B, and C, are all routing.

Of course my end goal is just B doing the routing.

A, B, and C all have SVIs with IP addresses for the x.110.215.16/28.

Basically, all of are vlans in the switches network, have SVIs on A, B, and C for each network.

I don't see anything pointing directly to SwitchC from SwitchB