VLAN Routing Problem

[sgraham]

Hi, I have a strange problem that I could do with some help on....

Setup....

2x 6500's with 1x 7206 and 1x MSFC2 router acting as VLAN routers.

5 Vlans with EIGRP.

The primary router currently is the default gateway and everything works fine, all networks are reachable.

The problem...

I wish to setup the second router as the HSRP router for the same VLAN's and for the most part this works fine. All the VLANS are mirrored addresses are all correct. Everything is working apart from one VLAN on the second router VLAN5, I setup the others no problem all can route, HSRP is working fine.

This VLAN however has a problem. The backup VLAN 5 ip address is not able to be reached from anywhere except on router itself, the primary can not ping the IP address on the backup, its almost as if its not visable. The config looks fine as the other VLANs are working on the backup and HSRP is functioning fine.

Any ideas?

[alfredshum]

It would be good if you can post the relevant configurations less any password

[sgraham]

Thanks for looking, here are the two router configs, the 7200 is trunked via 100full connection to a 6500.

[Kevin Dorrell]

The only thing I notice different about VLAN 5 is that standby 5 preempt is missing on the primary.

Edit : sorry I take that back - VLAN 20 has that difference as well. Is VLAN 20 working OK?

Kevin Dorrell

Luxembourg

[sgraham]

yes all other Vlans work fine with or without the standby commands. If place a PC on the same subnet as VLAN5 I can only ping the primary routers address, the ip address on the VLAN5 interface of the MSFC router is not pingable. All the other IP addresses for the other VLANs work fine though.

[Kevin Dorrell]

Does VLAN 5 actually exist on that 6500? I presume it is all part of a VTP domain. I remember that the VLAN interface is not marked as up unless there are actually active ports in the switch, which means that the VLAN must be actiove or trunked on the switch at layer 5. Now, I don't know what happens if you don't have any ports handling the VLAN (not even trunks), and the layer-3 interface is in standby.

What happens if you do a show int vlan 5 on the standby switch? How about a show vlan 5?

Kevin Dorrell

Luxembourg

[sgraham]

yes VLAN 5 is shown on the switch and is saying up on the router, to make sure the VTP was correct I put a port on the switch with the MSFC card into VLAN5

5 VLAN0005 active 9 3/19

10 VLAN0010 active 60

15 VLAN0015 active 61

Vlan5 is up, line protocol is up

Hardware is Cat6k RP Virtual Ethernet, address is 0008.7cd2.5c42 (bia 0008.7cd

2.5c42)

Description: Server VLAN 5

Internet address is 172.20.152.252/21

Its very confusing as all the other VLANs work correctly, if I create a new one it works fine.

[Kevin Dorrell]

I have been thinking about this problem overnight, but I still cannot work out what is going on.

What I would do as a next step is to take a PC configured in the subnet of VLAN 5, and attach it directly to a VLAN 5 port on the 6500, and then see if it can ping. That would distiguish whether it is the layer-3 VLAN interface that is not responding, or whether it is a case of VLAN 5 not being trunked correctly - e.g. pruned from a trunk.

Another thing you could look at to check the layer-2 connectivity is the spanning-tree for VLAN 5. If you look at it from the 6500 and from another switch in the network (perhaps the one you were testing from previously), do they have a consistent view of who is root for VLAN 5?

Are all the switches correctly in the same VTP domain, with the same configuration revision number?

Sorry, not solutions yet, but some diagnostic directions. Hope it helps.

Kevin Dorrell

Luxembourg

[sgraham]

Hi Kevin, ok did some more investigating, I connected a laptop directly into the 6506 switch with the MSFC2 card that has the problem, set the port to VLAN5 and tried pinging, it couldn't reach anything on either the same subnet or the gateway address for VLAN 5. I then changed the port to VLAN 15 one that works and had fully connectivty.

Below are two outputs from both core switches, the 6506 is the one that has the MSFC card, the 6509 is an existing switch with no problems. The VTP information looks ok.

6509-core> (enable) sho spantr 5

VLAN 5

Spanning tree mode PVST+

Spanning tree type ieee

Spanning tree enabled

Designated Root 00-02-fc-49-fc-04

Designated Root Priority 32768

Designated Root Cost 19

Designated Root Port 5/47

Root Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec

Bridge ID MAC ADDR 00-d0-01-2e-4c-04

Bridge ID Priority 32768

Bridge Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec

6506-Core> (Enable) sho spantr 5

VLAN 5

Spanning tree mode PVST+

Spanning tree type ieee

Spanning tree enabled

Designated Root 00-02-fc-49-fc-04

Designated Root Priority 32768

Designated Root Cost 0

Designated Root Port 1/0

Root Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec

Bridge ID MAC ADDR 00-02-fc-49-fc-04

Bridge ID Priority 32768

Bridge Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec

Port Vlan Port-State Cost Prio Portfast Channel_id

------------------------ ---- ------------- --------- ---- -------- ----------

3/1 5 forwarding 19 32 disabled 0

3/19 5 not-connected 19 32 disabled 0

15/1

[Kevin Dorrell]

OK, that should be useful information. Sorry if this post is an disconnected set of ramblings - trying to brainstorm here.

It points to a layer-2 problem, but the Spanning tree seems to work OK on VLAN 5. Furthermore, the 6506 is the root of the VLAN 5 Spanning Tree, and the 6509 knows it. So in theory everything should be working OK. Ummm ...

I take it that on the 6506, port 3/1 is a (trunk?) link to the 6509.

I see that there is not port listing on the 6509 for VLAN 5, but I presume that is because you have truncated the listing, right? And that port 5/47 on the 6509 is the (trunk?) link to the 6506, right?

I wonder if you have a unidirectional link ... no scrap that, because the other VLANs are working OK. Is this link 100BaseTX, or what?

Is there any chance you have a mismatched native VLAN across this link, with one end being VLAN 1 and the other end VLAN 5? I don't think so, thinking back to your earlier postings.

But the laptop on a VLAN 5 port on the 6506 cannot even ping the VLAN interface of the 6506. So the problem is more likely to be local in the 6506.

Do you have ip routing enabled on the 6506? Do you see anything strange in its routing table, like a missing entry for the VLAN 5 subnet?

I'll carry on thinking about it and let you know if anything occurs to me.

Kevin Dorrell

Luxembourg

[sgraham]

just checked a few things to answer you questions....

Yes 3/1 is 100TX trunk link and 5/47 is the trunk port on the other end.

Checked the native VLAN both end and they match.

The RSM MSFC2 card is inside teh 6506 and works for all the other VLANs except 5, it is sharing EIGRP information fine with the other routers.

One thing I just tried was to shut down the VLAN 5 interface in the MSFC2 router as soon as I do that VLAN 5 works receiving its router from the 7200. As soon as I bring the Interface back up it stops working. If I look at the Router within the 6506 the entry is there

C 172.20.152.0 is directly connected, Vlan5

[Kevin Dorrell]

It's as if both routers believe they are the active router for the standby group, and are fighting each other. Have you made the preempt consistent on both sides yet?

Do you get anything interesting from a show standby vlan 5 on each side?

Kevin Dorrell

Luxembourg

[sgraham]

HSRP output - you can see that they dont seem to know about each other. Neither router can ping the others 152.x address.

7200 Router

FastEthernet2/0.5 - Group 5

Local state is Active, priority 100, may preempt

Hellotime 3 holdtime 10

Next hello sent in 00:00:00.256

Hot standby IP address is 172.20.152.254 configured

Active router is local

Standby router is unknown expired

Standby virtual mac address is 0000.0c07.ac05

MSFC2 Router

Vlan5 - Group 5

Local state is Active, priority 110, may preempt

Hellotime 3 sec, holdtime 10 sec

Next hello sent in 2.444

Virtual IP address is 172.20.152.254 configured

Active router is local

Standby router is unknown

Virtual mac address is 0000.0c07.ac05

5 state changes, last state change 01:48:56

Same routers working VLAN example.................

MSFC2 Router

Vlan10 - Group 10

Local state is Active, priority 110, may preempt

Hellotime 3 sec, holdtime 10 sec

Next hello sent in 1.108

Virtual IP address is 172.20.128.254 configured

Active router is local

Standby router is 172.20.128.253 expires in 7.508

Virtual mac address is 0000.0c07.ac0a

7200 Router

FastEthernet2/0.10 - Group 10

Local state is Standby, priority 100, may preempt

Hellotime 3 holdtime 10

Next hello sent in 00:00:01.606

Hot standby IP address is 172.20.128.254 configured

Active router is 172.20.128.247 expires in 00:00:08

Standby router is local

[Kevin Dorrell]

It looks like a layer-2 problem then, doesn't it. The HSRP hellos should be multicast on 224.0.0.2, which corresponds to MAC 01:00:5e:00:00:02. You could try debug standby packe detail on each side to see if the Hellos are getting through but being ignored for some reason. It's RFC2281:

http://www.ietf.org/rfc/rfc2281.txt?number=2281

I'm afraid I'm running out of ideas. Sorry.

Address conflict on 172.20.152.252? Try shutting down the VLAN 5 interface on the MSFC2, then ping .252. I'm clutching at straws here!

Kevin Dorrell

Luxembourg

[sgraham]

checked the debug you see the hello out but not hello in foe the VLAN5, all the other VLANs have a hello out and in.

Out of interest I created a new VLAN on both routers, and it worked fine!!

I shut VLAN5 down to check there is no ip conflict, the IP is definatley not in use.

Very strange.