catalyst 3560 ipv6 bgp: route received, but not added to routing table

jperz · ‎01-16-2011

hi community,

I have a really weird issue with my cisco catalyst 3560G. After having researched back and forth, I'm running out of ideas ... maybe someone in here is guru enough to help me out!

device: cisco catalyst 3560g

ios: 12.2-53(SE) IP SERVICES

scenario: configured IPv6 iBGP link between this switch and a router. the router is announcing 1 (test) route. the switch sees the route properly, but doesn't add it to routing table (the route never gets to be "best path", although it should be).

here's a config snippet:

!

ipv6 unicast-routing

!

interface Vlan311

ip address 10.0.0.1 255.255.255.224

ip access-group 311-out out

no ip redirects

no ip unreachables

no ip proxy-arp

ipv6 address 2001:65F:E1:311::1/64

ipv6 enable

!

router bgp 12345

bgp router-id 10.9.1.1

bgp log-neighbor-changes

neighbor 2001:65F:E1:311::2 remote-as 12345

!

address-family ipv4

no neighbor 2001:65F:E1:311::2 activate

exit-address-family

!

address-family ipv6

neighbor 2001:65F:E1:311::2 activate

network 2001:65F:E1::/48

no synchronization

exit-address-family

!

the session comes up fine, but note that the path is missing the ">" marker for "best path":

sw01#show bgp ipv6 unicast neighbors 2001:65F:E1:311::2 routes

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path

* i2001:1638::/32 2001:65F:E1:311::2

100 0 6939 51058 i

and finally, here's what "debug" has to add to this:

Jan 16 19:15:27: IPv6RT[FFFF] RIB lookup failed

Jan 16 19:15:27: BGP(1): no valid path for 2001:1638::/32

The peer (2001:65F:E1:311::2) as well as its link-local ip is pingable.

Any hints?

thanks,

jakob

Peter Paluch · ‎01-16-2011

Hello Jakob,

You have an interesting issue. Can you please answer a few more questions?

I see you have the ipv6 unicast-routing on your 3560G. Do you also have the ip routing? I believe you have but just to be sure here.
Are you using the proper SDM template for IPv6 routing?
What does the show ip bgp ipv6 unicast 2001:1638::/32 say? Can you paste the output here?
What debug options did you use to produce the debug output you have enclosed with your original post?

Thank you!

Best regards,

Peter

jperz · ‎01-16-2011

hi,

you're right, i forgot to add a few details ...

1) yes, ip routing is on and working fine

2) i'm using "desktop IPv4 and IPv6 routing" template

3) output of show ip bgp ipv6 unicast 2001:1638::/32:

BGP routing table entry for 2001:1638::/32, version 0
Paths: (1 available, no best path)
Not advertised to any peer
6939 51058
2001:65F:E1:311::2 (FE80::20C:42FF:FE54:EFA8) (inaccessible) from 2001:65F:E1:311::2 (10.0.0.2)
Origin IGP, localpref 100, valid, internal
4) i used debug bgp ipv6 unicast and debug bgp ipv6 unicast updates

OK, seems we're getting somewhere, as the link-local address is marked inaccessible (guess I missed that before!). OTOH, the link-local ip is pingable:

sw01#ping ipv6 FE80::20C:42FF:FE54:EFA8
Output Interface: vlan311
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::20C:42FF:FE54:EFA8, timeout is 2 seconds:
Packet sent with a source address of FE80::217:EFF:FE4C:4447
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/8 ms

... and here's

sw01#show ipv6 neighbors vlan 311
IPv6 Address                              Age Link-layer Addr State Interface
2001:65F:E1:311::2                          0 000c.4254.efa8 REACH Vl311
FE80::20C:42FF:FE54:EFA8                    0 000c.4254.efa8 REACH Vl311

seems like a link-local problem?

=> enabling debug ipv6 nd

log:

Jan 16 23:47:52: ICMPv6-ND: Received NS for FE80::217:EFF:FE4C:4447 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:48:22: ICMPv6-ND: Received RA from FE80::20C:42FF:FE54:EFA8 on Vlan311
Jan 16 23:48:30: ICMPv6-ND: REACH -> STALE: FE80::20C:42FF:FE54:EFA8

Jan 16 23:50:18: ICMPv6-ND: Received NS for 2001:65F:E1:311::1 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:18: ICMPv6-ND: STALE -> DELAY: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: DELAY -> PROBE: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: Sending NS for FE80::20C:42FF:FE54:EFA8 on Vlan311
Jan 16 23:50:23: ICMPv6-ND: Received NA for FE80::20C:42FF:FE54:EFA8 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: PROBE -> REACH: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:28: ICMPv6-ND: Received NS for FE80::217:EFF:FE4C:4447 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:51:06: ICMPv6-ND: REACH -> STALE: FE80::20C:42FF:FE54:EFA8

so, link-local ip seems to "flap" rather often.

thanks,

jakob

lav · ‎12-22-2011

I also have this problem with iBGP ipv6 routes on catalyst 4506e (IOS XE 03.02.02.SG). Here is one such ignored route:

#show ip bgp ipv6 unicast 2A03:CA00:CA00:867::/64

BGP routing table entry for 2A03:CA00:CA00:867::/64, version 0

Paths: (1 available, no best path)

Not advertised to any peer

Local

2A03:CA00:922::1:0 (FE80::1E6F:65FF:FE38:7170) (inaccessible) from 2A03:CA00:922::1:0 (213.187.127.105)

Origin incomplete, localpref 100, valid, internal

2A03:CA00:922::1:0 _is_ accessible as shown below:

#show ipv6 route 2A03:CA00:922::1:0

Routing entry for 2A03:CA00:922::/64

Known via "connected", distance 0, metric 0, type connected

Redistributing via ospf 1

Backup from "ospf 1 [110]"

Route count is 1/1, share count 0

Routing paths:

directly connected via Vlan922

Last updated 20:55:47 ago

#ping 2A03:CA00:922::1:0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 2A03:CA00:922::1:0, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 0/5/12 ms

lgijssel · ‎01-16-2011

There is insufficient info in your starting post to resolve the issue.

Please follow the troubleshooting steps in the document below.

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_route_adv

If this doesn't resolve the problem, please provide the output as shown in the flowchart under the section:

Troubleshooting Routes Missing from the Routing Table

regards,

Leo

jperz · ‎01-16-2011

hi,

I just did that, the flowchart is telling me to open a TAC case

here's how i reached that point:

path is in bgp table => learned from iBGP => no synchronization => next hop is valid and reachable => path isn't marked as received only (afaik)

note that this flowchart assumes a "best path". in my case, however, the path isn't getting marked "best" in the first place

lgijssel · ‎01-16-2011

I would not be too sure of that. Please verify your ipv6 neighbor states.

http://www.cisco.com/en/US/docs/ios/ios_xe/ipv6/configuration/guide/ip6-addrg_bsc_con_xe.html#wp1054618

What I also noticed was the peculiar use of ipv6 addresses. You should use the eui64 convention as in the example below:

http://www.cisco.com/en/US/tech/tk872/technologies_configuration_example09186a0080b4a32f.shtml#diag

Using this convention maps the link-local address and any network address to the same host part of the address thus reducing the required number of multicast addresses. Without a lab, I cannot test it but the fact that this is not the case in your config may be a source of trouble.

This would make your route unreachable by an invalid next-hop. This was also the reason why I advised you to check ipv6 neigbor state.

regards,

Leo

jperz · ‎01-17-2011

thanks for your input!

EUI-64 (or more precisely, SLAAC) is by no means a convention or requirement! why should it be "peculiar" to statically configure an address? for routers and servers, where ip addresses hardly every change, i'd like to have everything statically and manually assigned, makes so life much easier. also, SLAAC doesn't offer any real advantage here.

i have other routers online (linux-based, juniper, etc.) which aren't configured SLAAC yet add bgp routes to the routing table properly :-)

as far as i can tell, ND does work. i'm seeing both (global and link-local) addresses in REACH state. i can ping both successfully. only thing i notice is a somewhat frequent flapping of neighbor states from REACH to STALE and back.

I have no idea why bgp believes the peer's FE80 is inaccessible ...

cadet alain · ‎01-17-2011

Hi,

FE80::20C:42FF:FE54:EFA8

this is a SLAAC address indeed but the the 7th bit in first octet of mac address hasn't been flipped so it is not a modified eui-64 but simply a eui-64 but nonetheless it isn't a static one

Regards.

alain.

Don't forget to rate helpful posts.

Peter Paluch · ‎01-17-2011

Alain,

this is a SLAAC address indeed but the the 7th bit in first octet of mac address hasn't been flipped

Hmmm, are you sure about that? The modified EUI-64 here is 020C:42FF:FE54:EFA8. After removing the FF:FE word, we get 020C:4254:EFA8. The first byte is 0x02 having clearly the 7th bit set to 1, so it is flipped after all.

In any case, something very fishy is going on here with the BGP...

Best regards,

Peter

cadet alain · ‎01-17-2011

Hi Peter,

Hmmm, are you sure about that? The modified EUI-64 here is 020C:42FF:FE54:EFA8. After removing the FF:FE word, we get 020C:4254:EFA8. The first byte is 0x02 having clearly the 7th bit set to 1, so it is flipped after all.

You're right and I am ashamed having posted such a stupidity

Kind Regards.

Alain.

Don't forget to rate helpful posts.

Peter Paluch · ‎01-17-2011

Hello Alain,

Don't even mention it. A few months ago, I posted a question about PIM Register messages stating that they are strangely addressed - to the multicast address instead of the RP address. I even developed a large hypothesis why that is so. Only two days later I've realized that the Wireshark was feeding me with the IP address of the innermost multicast IP packet but the outer IP packet with the PIM Register message was addressed completely correctly, to the RP. You can imagine my feelings at that moment

Best regards,

Peter

Peter Paluch · ‎01-17-2011

Hello Chris,

This just does not make sense. I have concocted a VERY quick scenario similar to your, and I have retaken most of your configs. I have connected my 3560V2, IOS 12.2(55)SE IPSERVICESK9, to a 1841 router running 15.0(1)M4 ADVENTERPRISEK9 (Fa0/23 on the 3560, Fa0/0 on the router). The relevant configs are:

3560:

version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Switch
!
boot-start-marker
boot-end-marker
!
!
!
!
no aaa new-model
system mtu routing 1500
ip routing
!
!
ipv6 unicast-routing
!
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
interface FastEthernet0/1
!
interface FastEthernet0/2
!
interface FastEthernet0/3
!
interface FastEthernet0/4
!
interface FastEthernet0/5
!
interface FastEthernet0/6
!
interface FastEthernet0/7
!
interface FastEthernet0/8
!
interface FastEthernet0/9
!
interface FastEthernet0/10
!
interface FastEthernet0/11
!
interface FastEthernet0/12
!
interface FastEthernet0/13
!
interface FastEthernet0/14
!
interface FastEthernet0/15
!
interface FastEthernet0/16
!
interface FastEthernet0/17
!
interface FastEthernet0/18
!
interface FastEthernet0/19
!
interface FastEthernet0/20
!
interface FastEthernet0/21
!
interface FastEthernet0/22
!
interface FastEthernet0/23
switchport access vlan 311
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/24
!
interface GigabitEthernet0/1
!
interface GigabitEthernet0/2
!
interface Vlan1
no ip address
shutdown
!
interface Vlan311
ip address 10.0.0.1 255.255.255.224
no ip redirects
no ip unreachables
no ip proxy-arp
ipv6 address 2001:65F:E1:311::1/64
ipv6 enable
!
router bgp 12345
bgp router-id 10.9.1.1
bgp log-neighbor-changes
neighbor 2001:65F:E1:311::2 remote-as 12345
!
address-family ipv4
no neighbor 2001:65F:E1:311::2 activate
no auto-summary
no synchronization
exit-address-family
!
address-family ipv6
neighbor 2001:65F:E1:311::2 activate
network 2001:65F:E1::/48
no synchronization
exit-address-family
!
ip classless
ip http server
ip http secure-server
!
!
ip sla enable reaction-alerts
!
!
!
!
line con 0
line vty 5 15
!
end

1841:

version 15.0
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
!
!
!
dot11 syslog
ip source-route
!
!
!
!
ip cef
ipv6 unicast-routing
ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
license udi pid CISCO1841 sn FCZ140890CS
!
redundancy
!
!
controller DSL 0/1/0
!
!
!
!
!
!
!
!
interface Loopback10
no ip address
ipv6 address 2001:1638::1/32
!
!
interface FastEthernet0/0
no ip address
duplex auto
speed auto
ipv6 address 2001:65F:E1:311::2/64
!
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
!
interface Serial0/0/0
no ip address
shutdown
no fair-queue
clock rate 2000000
!
!
interface Serial0/0/1
no ip address
shutdown
clock rate 2000000
!
!
router bgp 12345
bgp router-id 10.9.2.2
bgp log-neighbor-changes
neighbor 2001:65F:E1:311::1 remote-as 12345
!
address-family ipv4
no synchronization
no neighbor 2001:65F:E1:311::1 activate
no auto-summary
exit-address-family
!
address-family ipv6
no synchronization
network 2001:1638::/32
neighbor 2001:65F:E1:311::1 activate
exit-address-family
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
!
!
!
!
!
!
!
control-plane
!
!
!
line con 0
line aux 0
line vty 0 4
login
!
scheduler allocate 20000 1000
end

I did not shorten the configs, just to be sure nothing is omitted - once again, nothing has been removed from these configs, they are posted as-are.

Now, on the 3560:

Switch#show ip bgp ipv6 unicast sum
BGP router identifier 10.9.1.1, local AS number 12345
BGP table version is 2, main routing table version 2
1 network entries using 141 bytes of memory
1 path entries using 76 bytes of memory
2/1 BGP path/bestpath attribute entries using 280 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 497 total bytes of memory
BGP activity 1/0 prefixes, 1/0 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer InQ OutQ Up/Down State/PfxRcd
2001:65F:E1:311::2
                4 12345      21      17        2    0    0 00:15:26        1
Switch#show ip bgp ipv6 unicast
BGP table version is 2, local router ID is 10.9.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i2001:1638::/32   2001:65F:E1:311::2
                                             0    100      0 i
Switch#show ip bgp ipv6 unicast 2001:1638::/32
BGP routing table entry for 2001:1638::/32, version 2
Paths: (1 available, best #1, table Default)
Not advertised to any peer
Local
    2001:65F:E1:311::2 from 2001:65F:E1:311::2 (10.9.2.2)
      Origin IGP, metric 0, localpref 100, valid, internal, best
Switch#show ipv6 route
IPv6 Routing Table - Default - 4 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, R - RIP, D - EIGRP, EX - EIGRP external
       ND - Neighbor Discovery
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
C   2001:65F:E1:311::/64 [0/0]
     via Vlan311, directly connected
L   2001:65F:E1:311::1/128 [0/0]
     via Vlan311, receive
B   2001:1638::/32 [200/0]
     via 2001:65F:E1:311::2
L   FF00::/8 [0/0]
     via Null0, receive
Switch#ping 2001:1638::1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:1638::1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/8 ms
Switch#show ipv6 neigh
IPv6 Address                              Age Link-layer Addr State Interface
2001:65F:E1:311::2                          0 ec44.7682.dc12 REACH Vl311

So, I have no problems in establishing the BGP routing and getting the route into the routing table. The question is, what have I done differently? Can you compare your configuration with mine very closely and try to discover the differences? (Note that I have given the router the BGP Router-ID 10.9.2.2, you have a different Router-ID - are you sure it is unique?)

Sorry for a lengthy post. And oh, by the way, are you able to test a different IOS version on your 3560?

Best regards,

Peter

lgijssel · ‎01-17-2011

Great work Peter! And intersting scenario.

OP has edited his 2nd posting with additional info, as you certainly will have seen. This shows the problem is very likely a broken ipv6 neigbor state.

This leads BGP to set the next-hop as invalid. As there was little info on the router side, it remains guesswork to the exact cause of this.

Two things I noticed are:

1: a reference to 10.0.0.2 in the unreachable message.

Where is 10.0.0.2? (Router side BGP router-id?) This may conflict with 10.0.0.1/28 on the 3560.

2: The ip access group out on vlan 311. (Can we presume this is an ipv4 acl?)

Could this perhaps affect ipv6 operation? At least it is one of the things missing in your testconfig.

We have no info on how the acl looks.

regards,

Leo

jperz · ‎01-17-2011

Peter, Leo,

once again, thanks for your input!

I don't notice any obvious difference from your test config to mine.

I'm sure my router-id is unique

I can't reboot this switch because it's in production use, unfortunately. But I agree, could be problem where a single reboot magically resolves everything

the ipv4 address in brackets in the route output is the (correct) ipv4 address of the bgp peer

I also temporarily removed the ipv4 access-group on vlan311, but didn't change a thing either.

my bgp state is fine, here's output from show bgp ipv6 unicast neighbor:

fsw01#show bgp ipv6 unic nei 2001:65F:E1:311::2

BGP neighbor is 2001:65F:E1:311::2, remote AS 12345, internal link

Description: rtr01

BGP version 4, remote router ID 10.0.0.2

BGP state = Established, up for 00:01:25

Last read 00:00:29, last write 00:00:25, hold time is 180, keepalive interval is 60 seconds

Neighbor capabilities:

Route refresh: advertised and received(new)

Address family IPv6 Unicast: advertised and received

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 16 16

Notifications: 0 0

Updates: 17 19499

Keepalives: 5775 6623

Route Refresh: 0 1

Total: 5808 26139

Default minimum time between advertisement runs is 0 seconds

For address family: IPv6 Unicast

BGP table version 4, neighbor version 4/0

Output queue size : 0

Index 2, Offset 0, Mask 0x4

2 update-group member

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 1 1 (Consumes 76 bytes)

Prefixes Total: 1 1

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 0

Used as multipath: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

Total: 0 0

Number of NLRIs in the update sent: max 1, min 1

Address tracking is disabled

Connections established 16; dropped 15

Last reset 00:01:44, due to User reset

Transport(tcp) path-mtu-discovery is enabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Mininum incoming TTL 0, Outgoing TTL 255

Local host: 2001:65F:E1:311::1, Local port: 14569

Foreign host: 2001:65F:E1:311::2, Foreign port: 179

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x56A6BEB7C):

Timer Starts Wakeups Next

Retrans 5 0 0x0

TimeWait 0 0 0x0

AckHold 5 3 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 1 0 0x56A734E49

DeadWait 0 0 0x0

iss: 1891638869 snduna: 1891639047 sndnxt: 1891639047 sndwnd: 5760

irs: 3937504546 rcvnxt: 3937504737 rcvwnd: 16194 delrcvwnd: 190

SRTT: 146 ms, RTTO: 1283 ms, RTV: 1137 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms

Flags: higher precedence, nagle, path mtu capable

Datagrams (max data segment is 1440 bytes):

Rcvd: 10 (out of order: 0), with data: 5, total data bytes: 190

Sent: 11 (retransmit: 0), with data: 11, total data bytes: 644