eigrp update missing route between switches?

[Ryan Fisher]

So, I'm still learning eigrp and routing, so go easy on me!

This morning, I woke up to a bunch of emails that some remote systems were down.  After looking deeper into it, it appears that some routes aren't being exchanged between my core switch and my edge switch.

Topology:  two 6506 cores running vss, which connect to a 3750 edge switch, and run eigrp between them.  The 6506 is connected with a 3650 with hsrp for failover to the internet.

ED01-3750#sh ip route

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.201.1.2 to network 0.0.0.0

     172.16.0.0/24 is subnetted, 1 subnets

S       172.16.75.0 [1/0] via 10.74.193.1

     10.0.0.0/8 is variably subnetted, 52 subnets, 6 masks

D       10.13.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.3.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.199.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.13.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.3.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.13.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.3.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.1.201/32 [90/1536] via 10.200.253.2, 3w0d, Port-channel1

D EX    10.0.0.0/8 [170/145800448] via 10.200.253.2, 03:44:21, Port-channel1

D       10.200.1.200/32 [90/1536] via 10.200.253.2, 3w0d, Port-channel1

D       10.1.0.0/16 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.3.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.13.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.5.2.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.5.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.21.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.20.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.22.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.19.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.31.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.20.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.21.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.22.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.19.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.19.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.22.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.31.15.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.21.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.20.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.19.5.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.22.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.21.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.31.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.20.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D EX    10.32.0.0/16

           [170/256256] via 10.200.253.66, 03:45:24, GigabitEthernet2/0/14

C       10.200.253.0/30 is directly connected, Port-channel1

S       10.75.0.0/16 [1/0] via 10.200.253.74

D       10.100.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.100.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

C       10.200.253.72/29 is directly connected, GigabitEthernet2/0/21

C       10.200.253.64/29 is directly connected, GigabitEthernet2/0/14

S       10.74.193.0/24 [1/0] via 10.200.253.74

S       10.150.10.0/24 [1/0] via 10.201.1.2

D       10.200.2.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.202.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.3.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

C       10.201.1.0/24 is directly connected, Vlan511

D       10.200.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.205.1.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.10.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.1.9/32 [90/1536] via 10.200.253.2, 3w0d, Port-channel1

D       10.200.20.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

     198.xx.xx.0/30 is subnetted, 2 subnets

D       198.xx.xx.x [90/429312] via 10.200.253.66, 3w0d, GigabitEthernet2/0/14

D EX    198.xx.xx.x

           [170/256256] via 10.200.253.66, 3w0d, GigabitEthernet2/0/14

D    192.168.100.0/24 [90/3072] via 10.200.253.2, 3w0d, Port-channel1

S*   0.0.0.0/0 [1/0] via 10.201.1.2

VCORE-6506#sh ip route eigrp

     10.0.0.0/8 is variably subnetted, 51 subnets, 6 masks

D EX    10.32.0.0/16 [170/256512] via 10.200.253.1, 03:43:06, Port-channel100

D EX    10.0.0.0/8 [170/145800192] via 10.200.199.3, 03:43:12, Vlan999

D       10.200.253.72/29 [90/28416] via 10.200.253.1, 3d20h, Port-channel100

     198.18.35.0/30 is subnetted, 2 subnets

D       198.18.35.4 [90/429568] via 10.200.253.1, 3w0d, Port-channel100

D EX    198.18.35.8 [170/256512] via 10.200.253.1, 3w0d, Port-channel100

D*EX 0.0.0.0/0 [170/256256] via 10.200.253.1, 7w0d, Port-channel100

So, the networks I'm looking for on the core switch are 10.75.0.0 and 10.74.193.0, of which are not being propagated from the edge switch to the core.  When I put those routes in as static on the core and point them to the edge switch as the next hop, the networks come back and everything works.  When I remove them, the stop working again.

When I do a traceroute from a machine to something on the network, it tries to go to the hsrp switch member with the core.  That's weird, since the gateway of last resort on the core should route those requests to the edge switch, I would think.

[root@rmf-fs01-lx ~]# traceroute 10.75.199.1

traceroute to 10.75.199.1 (10.75.199.1), 30 hops max, 40 byte packets

1  10.200.1.1 (10.200.1.1)  1.363 ms  1.345 ms  1.359 ms

2  10.200.199.3 (10.200.199.3)  0.676 ms  0.785 ms  0.996 ms

3  * * *

4  * * *

When this happened this morning, access to another one of my remote sites went down also (that's connected to that edge switch as well), but that one came back up again within a minute or so.  These other two networks, 10.75.0.0 and 10.74.193.0 didn't come back up and continue to be down. (unless I put those static routes in).  A sh log on both switches shows nothing.

Anyone have any ideas?  I tried to provide enough information upfront in hopes someone can help because I'm at a loss, and because I'm still trying to learn eigrp, my knowledge is limited.  My thought is that I could force an eigrp update between the two switches, but from what I've read it doesn't seem like a good idea to try to force anything.

Thanks!

[Richard Burts]

Ryan

You tell us that the edge switch is running EIGRP with the core. But you do not tell us how the edge switch is configured. In particular you do not tell us whether the edge switch is redistributing static routes into EIGRP.

I see in the route table of the edge switch that the 2 networks you mention are in the route table as static routes. If the edge switch is not redistributing static then that would explain why the core is not learning the routes.

HTH

Rick

[Ryan Fisher]

thanks for the response.  How do I tell if it's configured to redistribute static routes?  What baffles me is that it was working fine for about the last month and all of a sudden it quit working this morning.  (along with that other network to the remote site that came back up)

here's the config for the edge switch

router eigrp 100

network 0.0.0.0

redistribute static metric 10000 0 255 1 1500 route-map STATIC>EIGRP

passive-interface default

no passive-interface Port-channel1

no passive-interface GigabitEthernet2/0/14

!

ip classless

ip route 0.0.0.0 0.0.0.0 10.201.1.2 name DEFAULT-TO-ASA

ip route 10.74.193.0 255.255.255.0 10.200.253.74 name to_DR_net

ip route 10.75.0.0 255.255.0.0 10.200.253.74 name to_DR_Net

ip route 10.150.10.0 255.255.255.0 10.201.1.2

ip route 172.16.75.0 255.255.255.0 10.74.193.1

ok, duh, after looking at the config I see the line "redistribute static" which must mean that.  (This was set up by our Cisco integrator, and I've been trying to learn what he's done).  So it looks like it's supposed to be doing that.

here's the config for the core:

router eigrp 100

network 0.0.0.0

no auto-summary

!

ip classless

ip route 10.13.253.0 255.255.255.248 10.200.199.3 name STATIC-TO-HSRP-DR-SWITCH

ip route 10.150.10.0 255.255.255.0 10.200.253.1 name VPN

ip route 10.200.253.64 255.255.255.248 10.200.253.1 name ROUTE-TO-WAN-EDGE

ip route 10.201.1.0 255.255.255.0 10.200.253.1 name ROUTE-TO-ASA-ADMIN

There's no redistribute static on the core because technically the place there needs to be static routes are on the edge switch.  (I would guess)  Does this help at all?

Thanks!

[Richard Burts]

Ryan

Yes the information provided in this post does show that the edge switch is configured to redistribute static routes, but that is controlled by the route map STATIC>EIGRP. So what is configured in that route map?

Also is it possible that anyone made any config changes this morning? If it was working and stopped working this morning then it seems logical that either someone made a change this morning or that there was some network event that impacted things.

In your original post you gave the route table from the edge switch. Was this taken while the problem was actively being experienced or is this from a time when the network was working ok?

HTH

Rick

[Peter Paluch]

Hello,

In addition to Rick's spot-on comments, it would be very interesting to see the output of these commands both on the core and on the edge switch:

• show ip protocols
• show ip eigrp neighbor
• show ip eigrp topology

Especially the "show ip eigrp topology" should reveal whether the redistributed static routes are present in the EIGRP's topology database - the main data storage of EIGRP. If they are not present there (especially on the edge switch), then the redistribution is not working as expected.

Best regards,

Peter

[Ryan Fisher]

Thanks for the continued help.  Here's what's in the map:

route-map STATIC>EIGRP permit 10

match ip address prefix-list STATIC>EIGRP

Information from edge switch:

ED01-3750#sh ip protocols

*** IP Routing is NSF aware ***

Routing Protocol is "eigrp 100"

  Outgoing update filter list for all interfaces is not set

  Incoming update filter list for all interfaces is not set

  Default networks flagged in outgoing updates

  Default networks accepted from incoming updates

  Redistributing: static, eigrp 100

  EIGRP-IPv4 Protocol for AS(100)

    Metric weight K1=1, K2=0, K3=1, K4=0, K5=0

    NSF-aware route hold timer is 240

  EIGRP NSF disabled

     NSF signal timer is 20s

     NSF converge timer is 120s

    Router-ID: 10.201.1.1

    Topology : 0 (base)

      Active Timer: 3 min

      Distance: internal 90 external 170

      Maximum path: 4

      Maximum hopcount 100

      Maximum metric variance 1

  Automatic Summarization: disabled

  Maximum path: 4

  Routing for Networks:

    0.0.0.0

  Passive Interface(s):

    Vlan511

  Routing Information Sources:

    Gateway         Distance      Last Update

    (this router)         90      28w1d

    10.200.253.2          90      05:26:34

    10.200.253.66         90      05:26:34

    10.201.1.5            90      25w1d

  Distance: internal 90 external 170

ED01-3750#sh ip eigrp neighbors

EIGRP-IPv4 Neighbors for AS(100)

H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq

                                            (sec)         (ms)       Cnt Num

1   10.200.253.66           Gi2/0/14          11 3w0d        1   200  0  487

0   10.200.253.2            Po1               12 28w6d       3   200  0  925

ED01-3750#sh ip eigrp topology

EIGRP-IPv4 Topology Table for AS(100)/ID(10.201.1.1)

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

       r - reply Status, s - sia Status

P 10.22.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.13.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.1.9/32, 1 successors, FD is 1536

        via 10.200.253.2 (1536/256), Port-channel1

P 10.202.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.19.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.3.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.32.0.0/16, 1 successors, FD is 256256, tag is 4323

        via 10.200.253.66 (256256/256000), GigabitEthernet2/0/14

P 10.20.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.13.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.5.2.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.1.0.0/16, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.19.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 198.xx.xx.x/30, 1 successors, FD is 256256

        via 10.200.253.66 (256256/256000), GigabitEthernet2/0/14

P 192.168.100.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.253.0/30, 1 successors, FD is 1536

        via Connected, Port-channel1

P 10.0.0.0/8, 1 successors, FD is 145800448

        via 10.200.253.2 (145800448/145800192), Port-channel1

P 10.31.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.31.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.21.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.2.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.205.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.3.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.5.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.201.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan511

P 10.100.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.3.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.20.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.253.64/29, 1 successors, FD is 2816

        via Connected, GigabitEthernet2/0/14

P 0.0.0.0/0, 1 successors, FD is 256000

        via Rstatic (256000/0)

P 10.22.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.13.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.21.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.19.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.1.201/32, 1 successors, FD is 1536

        via 10.200.253.2 (1536/256), Port-channel1

P 10.21.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.20.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.31.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.21.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 198.18.35.4/30, 1 successors, FD is 429312

        via 10.200.253.66 (429312/429056), GigabitEthernet2/0/14

P 10.100.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.19.5.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.199.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.3.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.22.10.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.253.72/29, 1 successors, FD is 28160

        via Connected, GigabitEthernet2/0/21

P 10.13.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.20.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.20.15.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.200.1.200/32, 1 successors, FD is 1536

        via 10.200.253.2 (1536/256), Port-channel1

P 10.150.10.0/24, 1 successors, FD is 256000

        via Rstatic (256000/0)

P 10.22.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

P 10.3.1.0/24, 1 successors, FD is 3072

        via 10.200.253.2 (3072/2816), Port-channel1

Information from Core switch:

VCORE-6506#sh ip protocols

*** IP Routing is NSF aware ***

Routing Protocol is "eigrp 100"

  Outgoing update filter list for all interfaces is not set

  Incoming update filter list for all interfaces is not set

  Default networks flagged in outgoing updates

  Default networks accepted from incoming updates

  EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0

  EIGRP maximum hopcount 100

  EIGRP maximum metric variance 1

  Redistributing: eigrp 100

  EIGRP NSF-aware route hold timer is 240s

  EIGRP NSF disabled

     NSF signal timer is 20s

     NSF converge timer is 120s

  Automatic network summarization is not in effect

  Maximum path: 4

  Routing for Networks:

    0.0.0.0

  Routing Information Sources:

    Gateway         Distance      Last Update

    10.200.199.3          90      05:30:31

    10.200.253.1          90      05:30:31

  Distance: internal 90 external 170

SDVCORE-6506#sh ip eigrp neighbors

EIGRP-IPv4 neighbors for process 100

H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq

                                            (sec)         (ms)       Cnt Num

1   10.200.199.3            Vl999             12 23w3d       4   200  0  786

0   10.200.253.1            Po100             10 28w6d       6   200  0  1100

VCORE-6506#sh ip eigrp topology

EIGRP-IPv4 Topology Table for AS(100)/ID(192.168.100.1)

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

       r - reply Status, s - sia Status

P 0.0.0.0/0, 1 successors, FD is 256256

        via 10.200.253.1 (256256/256000), Port-channel100

P 10.13.5.0/24, 1 successors, FD is 2816

        via Connected, Vlan305

P 10.3.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan210

P 10.200.199.0/24, 1 successors, FD is 2816

        via Connected, Vlan999

P 10.13.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan301

P 10.3.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan215

P 10.13.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan315

P 10.3.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan201

P 10.0.0.0/8, 1 successors, FD is 145800192

        via 10.200.199.3 (145800192/145799936), Vlan999

P 10.200.1.201/32, 1 successors, FD is 256

        via Rstatic (256/0)

P 10.1.0.0/16, 1 successors, FD is 2816

        via Connected, Vlan10

P 10.200.1.200/32, 1 successors, FD is 256

        via Rstatic (256/0)

P 192.168.100.0/24, 1 successors, FD is 2816

        via Connected, Vlan903

P 10.3.5.0/24, 1 successors, FD is 2816

        via Connected, Vlan205

P 10.5.2.0/24, 1 successors, FD is 2816

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

       r - reply Status, s - sia Status

        via Connected, Vlan51

P 10.13.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan310

P 10.5.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan50

P 10.21.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan715

P 10.20.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan615

P 10.22.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan815

P 10.19.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan410

P 10.31.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan901

P 10.20.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan610

P 10.21.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan710

P 10.22.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan810

P 10.19.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan415

P 10.19.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan401

P 10.22.5.0/24, 1 successors, FD is 2816

        via Connected, Vlan805

P 10.31.15.0/24, 1 successors, FD is 2816

        via Connected, Vlan915

P 10.21.5.0/24, 1 successors, FD is 2816

        via Connected, Vlan705

P 10.20.5.0/24, 1 successors, FD is 2816

        via Connected, Vlan605

P 10.19.5.0/24, 1 successors, FD is 2816

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

       r - reply Status, s - sia Status

        via Connected, Vlan405

P 10.22.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan801

P 10.21.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan701

P 10.31.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan910

P 10.20.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan601

P 10.32.0.0/16, 1 successors, FD is 256512, tag is 4323

        via 10.200.253.1 (256512/256256), Port-channel100

P 10.200.253.0/30, 1 successors, FD is 1536

        via Connected, Port-channel100

P 10.100.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan510

P 10.100.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan600

P 10.200.253.72/29, 1 successors, FD is 28416

        via 10.200.253.1 (28416/28160), Port-channel100

P 10.200.253.64/29, 0 successors, FD is Inaccessible

        via 10.200.253.1 (3072/2816), Port-channel100

P 10.150.10.0/24, 0 successors, FD is Inaccessible

        via 10.200.253.1 (256256/256000), Port-channel100

P 10.200.2.0/24, 1 successors, FD is 2816

        via Connected, Vlan502

P 10.202.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan521

P 10.200.3.0/24, 1 successors, FD is 2816

        via Connected, Vlan503

P 10.201.1.0/24, 0 successors, FD is Inaccessible

        via 10.200.253.1 (3072/2816), Port-channel100

P 10.200.1.0/24, 1 successors, FD is 2816

        via Connected, Vlan501

P 10.205.1.0/24, 1 successors, FD is 2816

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

       r - reply Status, s - sia Status

        via Connected, Vlan505

P 10.200.10.0/24, 1 successors, FD is 2816

        via Connected, Vlan110

P 10.200.1.9/32, 1 successors, FD is 256

        via Rstatic (256/0)

P 10.200.20.0/24, 1 successors, FD is 2816

        via Connected, Vlan120

P 198.xx.xx.x/30, 1 successors, FD is 429568

        via 10.200.253.1 (429568/429312), Port-channel100

P 10.13.253.0/29, 0 successors, FD is Inaccessible

        via 10.200.199.3 (12825856/12825600), Vlan999

P 198.xx.xx.x/30, 1 successors, FD is 256512

        via 10.200.253.1 (256512/256256), Port-channel100

Looking at the topology, those networks aren't included in there, so I guess I don't know what to do next.  I can guarantee that no one else was in it, because we're pretty much a one-guy shop with me owning the network, and the only other person that has access to this is our Cisco integrator, and the project was finished, so I don't see any reason why he'd be in.  So, other than someone changing something, I don't really know what could've happened, since it was working fine up until 8:00 this morning.

The other network that went down and back up was the remote network 10.32.0.0.  That's connected to the router (connected to the edge switch) 10.200.253.66.  Thanks again!

[Peter Paluch]

Ryan,

The route-map references a prefix-list called STATIC>EIGRP - can you please post its configuration as well? Look for lines ip prefix-list STATIC>EIGRP in your configuration - we need to see all of them.

In any case, the static routes do not seem to be redistributed into your EIGRP process in this moment.

Best regards,

Peter

[Ryan Fisher]

Thanks for the reply. It probably got buried in the last post, but the static map was at the top of my last reply. Thanks!

[Peter Paluch]

Ryan,

Don't confuse the route-map called "STATIC>EIGRP" with a prefix-list of the same name. You have pasted the route-map but not the prefix-list. Please read my requests more carefully

Best regards,

Peter

[Ryan Fisher]

Oh, ok. I'm out on an errand right now, so I'll get that when I get back shortly. Thanks!

[Ryan Fisher]

ok, so these are the two lines with that configuration:

ip prefix-list STATIC>EIGRP seq 5 permit 0.0.0.0/0

ip prefix-list STATIC>EIGRP seq 10 permit 10.150.10.0/24

It looks like it's referencing the entire network?

So, if the static routes aren't getting redistributed with eigrp, what can I do to get them back in?  They must have been in before when it was working, but not since whatever happened this morning?

Thanks!

[Richard Burts]

Ryan

Thanks for the additional information. The first line of the prefix list is permitting redistribution of the default route and the second line permits redistribution of only a single specific network (10.150.10.0/24). No other static routes will be redistributed. So this means that the current configuration would not redistribute either 10.75.0.0 or 10.74.193.0 and therefore the edge switch would not advertise these networks to the core.

At first I thought that this was the problem. But as I examined the information available I realize that the default route on the core seems to have the edge switch as the next hop. Is that the case?

If the default route of the core does use the edge as the next hop, then the static routes from the edge router do not need to be advertised to the core since the default route on the core will send to the edge to get to those networks. So I believe that now the question becomes - why was the default route on the core not sending traffic for those networks to the edge and why did you need to add static routes to make things work?

Is it possible that there is a command in the core config for "ip classless" or for "no ip classless"?

HTH

Rick

[Ryan Fisher]

Thanks a lot for the reply.  Yes, on the core the gateway of last resort does go to the edge switch, so that makes sense the edge would not need to advertise those static routes to it.  So, I guess that's why this is so confusing, why these two networks are not routing to that edge switch, but routing to the hsrp partner to the core, 10.200.199.3, as you saw in the traceroute above.  And the weird thing, it all of a sudden just started happening this morning, and has been down ever since.

What's also weird, though, is that I got a bunch of emails at about 6:15 - 6:30 tonight, that made it look like the route started to work for a second, and then quit working again.  I get email alerts from my Compellent SAN when it can't replicate to the remote volumes, and since it's been down since this morning it basically tells you once when they're down and that's it.  Tonight at 6:15, I got more emails from it, which basically means that it connected again, then went down again, so that makes me think the route worked for a second, then quit again.  Man, I wish I knew what was happening!

Also, to answer your question, both the core switch and the edge switch has ip classless in its config.  Thanks a lot for sticking this out with me!

[Ryan Fisher]

ok, so believe it or not I got it working again.  I decided to do a reload on the the HSRP partner switch, the one that those networks kept trying to route to, and when it went down the routing got figured out and started working again!  (I'm running HSRP with the core to a 3650 on another floor that is supposed to route out to the internet in case we lose the data center where the core is.  This way, users can access the internet from their offices and potentially get to our DR site via citrix over the internet.  Management requirement...    )   Anyways, since no one uses that switch, I figured it couldn't hurt to reboot it.

So, that made it work again, but I kind of need to figure out how this happened so it doesn't happen again.  As I was typing the above paragraph, it got me thinking that maybe the core switching failed over to that 3650, and that gateway of last resort is the internet, and the gateway of last resort on the 6500 is the edge switch.  But, what I don't understand is, if that's what happened, why did the other network to my remote site that went down this morning come back, when these didn't?  These were the only two that were affected.  Weird.

Any insight is greatly appreciated.  Thanks!

[Richard Burts]

Ryan

I am glad that you got it working again. Here is my current best guess at the reason it happened. In the morning something happened that caused HSRP to make the 3650 the lead/active router and the core to be the standby. So traffic from devices at your site would forward to the 3650 rather than the core. I am guessing that the 3650 had routes to most of the subnets in the network but not to those two networks. And the 3650 has a default route that goes toward the Internet and not toward the 3750 as the core does. So traffic from PCs at your site would go through the 3650 and most of your networks work but these two fail.

Perhaps you can check the content of the routing table of the 3650 and verify whether it has most of the networks but not these two?

HTH

Rick