EIGRP Neighbor not on common subnet: packet loss

[mfcornett]

I was called into look at a customers's site that was upgraded by another vendor. They were complaining about packet loss and performance issues. I discovered they had changed from static routing and turned on EIGRP (100) and the EIGRP neighbors are not on a common subnet, they used secondary IP addressing to connect two 4506 switches to the core router. Correct me if I'm wrong, but won't this cause the neighbors to bounce and cause the degraded network performance??? Don't we need to have a single common subnet for this to work? Either that or they have bridged the VLANs somehow? There is also a cable connecting the 4506 switches besides the uplink to the core router.

4506#1

May 17 21:28:55.443: EIGRP-IPv4(100): Neighbor 192.168.120.1 not on common subnet for Vlan1

May 17 21:29:05.848: EIGRP-IPv4(100): Neighbor 10.102.10.10 not on common subnet for Vlan20

May 17 21:29:17.327: EIGRP-IPv4(100): Neighbor 206.78.xxx.xx not on common subnet for Vlan20

May 17 21:29:48.291: EIGRP-IPv4(100): Neighbor 206.78.xxx.x not on common subnet for Vlan30

!

interface Vlan1

ip address 192.168.122.10 255.255.255.0 secondary

ip address 192.168.121.10 255.255.255.0

no ip redirects

no ip route-cache cef

no ip route-cache

no ip mroute-cache

!

interface Vlan20

ip address 10.101.10.100 255.255.0.0

no ip route-cache cef

no ip route-cache

!

interface Vlan30

ip address 10.201.10.100 255.255.0.0

!

interface Vlan40

no ip address

!

ip route 0.0.0.0 0.0.0.0 192.168.121.1

!

4506 #2

May 17 22:22:49.471: EIGRP-IPv4(100): Neighbor 192.168.120.1 not on common subnet for Vlan1

May 17 22:23:10.579: EIGRP-IPv4(100): Neighbor 206.78.xxx.xx not on common subnet for Vlan20

May 17 22:22:59.775: EIGRP-IPv4(100): Neighbor 192.168.240.1 not on common subnet for Vlan40

!

interface Vlan1

ip address 192.168.121.11 255.255.255.0 secondary

ip address 192.168.122.11 255.255.255.0

no ip redirects

!

interface Vlan20

ip address 10.102.10.10 255.255.0.0

no ip route-cache cef

no ip route-cache

no ip mroute-cache

!

interface Vlan30

no ip address

!

interface Vlan40

ip address 192.168.242.10 255.255.255.0

!

ip route 0.0.0.0 0.0.0.0 192.168.122.1

!

Core Router:

!

interface GigabitEthernet0/1.1

encapsulation dot1Q 1 native

ip address 192.168.122.1 255.255.255.0 secondary

ip address 192.168.121.1 255.255.255.0 secondary

ip address 192.168.100.65 255.255.255.248 secondary

ip address 192.168.120.1 255.255.255.0

no snmp trap link-status

!

!

interface GigabitEthernet0/1.20

encapsulation dot1Q 20

ip address 192.168.4.193 255.255.255.248 secondary

ip address 192.168.2.97 255.255.255.224 secondary

ip address 10.100.0.1 255.255.0.0 secondary

ip address 10.101.0.1 255.255.0.0 secondary

ip address 10.102.0.1 255.255.0.0 secondary

ip address 192.168.100.129 255.255.255.224 secondary

ip address 206.78.xxx.xx 255.255.255.224

no snmp trap link-status

!

!

interface GigabitEthernet0/1.30

encapsulation dot1Q 30

ip address 10.200.0.1 255.255.0.0 secondary

ip address 10.201.0.1 255.255.0.0 secondary

ip address 10.202.0.1 255.255.0.0 secondary

ip address 206.78.xxx.x 255.255.255.0

!

interface GigabitEthernet0/1.40

description Guest WLAN Only

encapsulation dot1Q 40

ip address 192.168.241.1 255.255.255.0 secondary

ip address 192.168.242.1 255.255.255.0 secondary

ip address 192.168.240.1 255.255.255.0

!

Thanks for the help,

Mark

[Harold Ritter]

Hi Mark,

These messages are normal considering that the routers do not share a common primary subnet on each vlan. Please refer to the following document:

http://www.cisco.com/en/US/tech/tk365/technologies_q_and_a_item09186a008012dac4.shtml#three

Regards

[Julio Carvajal]

Hello Mark,

I agree with Harold,

Here is the thing, with EIGRP we are never going to source hello packets from secondary subnets so the neigbhor relationship will never be seen on those subnets as on one side the primary subnets are "X" and in the other side are "Y".

That being said, there is definetly an implementation problem here as there are mismatches on what are consider primary and secondary addresses, try to fix that and let us know how it goes.

Regards,

Julio Carvajal

[Richard Burts]

Mark

Whoever did the conversion to EIGRP certainly did a poor job of it. One result, as pointed out by Julio, is that EIGRP neighbor relationships would never form. And that means that EIGRP would never advertise anything. The neighbors would not bounce because they would never come up. Your original post said that there had been packet loss and performance problems. I do not believe that the EIGRP issues could cause packet loss, since EIGRP is really not doing anything in this environment. So the cause of packet loss and performance problems is somewhere else.

I see that both switches have a valid static default route configured which would forward traffic to the core router (on one of its secondary interfaces). You have not shown what routes exist on the core and I wonder if some of the problems might be that the core is not routing correctly to send some traffic back to the switches.

I also notice that switch 1 has disabled CEF on vlan 1 and vlan 20 and that switch 2 has disabled CEF on vlan 20. Disabling CEF certainly has potential to cause performance problems.

I suggest that the first step in addressing the issues would be to enable CEF on all interfaces where it is disabled. Then it would be beneficial to reconfigure interfaces so that there is consistency about which subnets are present on each interface and so that there is a single subnet which is primary on each vlan/subinterface.

If these issues are addressed and the problem is still present I would look into the possiblity that the switches communicate with the core router but not with each other. This could perhaps result in the core router forwarding packets for a subnet to switch 1 while the destination is really on switch 2.

HTH

Rick

[mfcornett]

Excellent responses gentlemen! I do believe you are on the right track, while I see EIGRP configured on the core router, they left in place the old static routes to legacy VLANS but there are no route entries for return trafic to the new switches. This was a rip and replace upgrade by another vendor at a local school district we support, and at one time we had everything running rather smoothly using a combination of Policy Based Routing and static routes.

We lost the bid on the upgrade and the installers had difficulty getting all traffic flowing so they told the client they were implementing EIGRP and adding new VLANS to make the network more efficient on the 10GB backbone, but in fact made matters even worse. I am going to do some trace routes, reconfigure the interfaces and add in the new static routes at the core router.

What I am seeing is periodic packet loss exceeding 50-60% on IDF switches as users come online resulting in a domino effect, or broadcast storm that eventually brings down the interfaces on the core router. This was not a clean install and is a rather complex network that is now tied up in knots.

Apparently they have a few servers with Public interfaces that are getting hammered by Hong Kong. China, etc... which is definately wreaking havoc on the network and causing users to lose connectivity to admin and student servers. We used to have PIX firewalls in the loop but they dumped them as well.

I hope to have this ironed out soon as they are upgrading their site-to-site wireless to two remote sites and adding even more traffic to the network backbone. Thanks for the help, I will let you know how it goes.

Here is some sample output from the core router:

!

router eigrp 100

network 10.0.0.0

network 192.168.1.0

network 192.168.2.0

network 192.168.100.0

network 192.168.120.0

network 192.168.121.0

network 192.168.122.0

network 192.168.240.0

network 192.168.241.0

network 192.168.242.0

no auto-summary

no eigrp log-neighbor-changes

!

!
ip route 0.0.0.0 0.0.0.0 192.168.100.205 90
ip route 10.103.0.0 255.255.0.0 192.168.100.70 90
ip route 10.103.0.0 255.255.0.0 192.168.100.10 100
ip route 10.104.0.0 255.255.0.0 192.168.120.25 90
ip route 10.104.0.0 255.255.0.0 192.168.100.14 100
ip route 10.105.0.0 255.255.128.0 192.168.100.6 90
ip route 10.105.128.0 255.255.128.0 192.168.100.18 90
ip route 10.106.0.0 255.255.255.0 192.168.100.22 90
ip route 10.203.0.0 255.255.0.0 192.168.100.70 90
ip route 10.203.0.0 255.255.0.0 192.168.100.10 100
ip route 10.204.0.0 255.255.0.0 192.168.120.25 90
ip route 10.204.0.0 255.255.0.0 192.168.100.14 100
ip route 10.205.0.0 255.255.128.0 192.168.100.6 90
ip route 10.205.128.0 255.255.128.0 192.168.100.18 90
ip route 172.20.20.128 255.255.255.224 192.168.100.6 90
ip route 172.30.1.0 255.255.255.0 192.168.100.201 90
ip route 192.168.100.96 255.255.255.240 192.168.100.6 90
ip route 192.168.100.112 255.255.255.240 192.168.100.6 90
ip route 204.xxx.0.0 255.255.255.0 192.168.100.201 90
ip route 204.xxx.4.0 255.255.255.0 192.168.100.201 90
ip route 206.78.xxx.xx 255.255.255.224 192.168.100.6 90
!

[Richard Burts]

Thanks for the additional information that does supply some context about what is happening in this network. It seems it will be quite a task to get it all straightened out. I would suggest that the first step should be to get consistent interface configurations between switches so that they agree on what subnets are on what interfaces and which subnet is primary and the others are secondary.

You could keep EIGRP running in the network and it should do ok after you straighten out the inconsistencies of what subnet is primary and what subnets are on what interfaces. Or you probably could go back to static roues and PBR. I do not have enough understanding of the network topology to have a good opinion about which would be better.

HTH

Rick

[Mark Graham]

would it be advisable for the moment to add another vlan interface on each device, add a non overlapping IP address scheme on that vlan, add the network into the routing table, then set each vlan that is getting neighbor errors passive?

it wont solve the problem, but the router wont try to form adjencies with those subnetworks if i'm correct.

[Richard Burts]

Creating a new VLAN, addressing the VLAN, running EIGRP on that VLAN, and making all other interfaces passive would allow EIGRP to run and to form neighbor relationships. But from my perspective it is doing work that will need to be undone as soon as the mismatching interface subnets are corrected and just slows down the work of fixing the real problems.

As currently configured it looks like EIGRP is not forming neighbor relationships and so is doing no good. I suggest that it might be more effective to remove EIGRP, depend on the static routes that are in place, fix the mismatching interface subnets, and put EIGRP back in.

HTH

Rick

[Mark Graham]

yeah, i do realise its just a bandaid to the issue richard    but it is something that would get him by untill he can take the network down for maintainence, but as you said, static routes its almost moot. 

[mfcornett]

Thanks for the help everyone! I have an appt. onsite tomorrow morning to take down the network to impliment the changes and will let you guys know how it goes. I added sanitized running-configs for DO core router, HS & MS core switches with notations. Also included is some output from sh ip eigrp accounting, sh ip eigrp neighbor, and sh ip eigrp topology. Looks like I have some routing issues to untangle and have already fixed a few problems, like removing overlapping LWAP DHCP scopes from the core switches that were also active on the WLC. Been lots of surprises so far. 

[Richard Burts]

Mark

Congratulations on having resolved some of the issues. And good luck in continuing to find and fix problems in this network. I have taken a quick look at the configs that you posted and I am a bit concerned. In the 3845 I find this interface configuration

interface GigabitEthernet0/1.1
encapsulation dot1Q 1 native
ip address 192.168.122.1 255.255.255.0 secondary ##Middle School
ip address 192.168.121.1 255.255.255.0 secondary ## High School
ip address 192.168.100.65 255.255.255.248 secondary ## Wireless Bridge
ip address 192.168.120.1 255.255.255.0 ## DO
no snmp trap link-status

in the config for HS MDF 4506 I find this

interface Vlan1
ip address 192.168.122.10 255.255.255.0 secondary
ip address 192.168.121.10 255.255.255.0

and in the config for MS MDF 4506 I find this

interface Vlan1
ip address 192.168.121.11 255.255.255.0 secondary
ip address 192.168.122.11 255.255.255.0

The good news is that both 4506s agree with which network/subnet is primary and which is secondary. They will be able to form EIGRP neighbor relationship. The not good news is that they do not agree with the 3845 which will not form EIGRP neighbor relationship with either 4506.

HTH

Rick