01-16-2011 11:18 AM - edited 03-04-2019 11:06 AM
hi community,
I have a really weird issue with my cisco catalyst 3560G. After having researched back and forth, I'm running out of ideas ... maybe someone in here is guru enough to help me out!
device: cisco catalyst 3560g
ios: 12.2-53(SE) IP SERVICES
scenario: configured IPv6 iBGP link between this switch and a router. the router is announcing 1 (test) route. the switch sees the route properly, but doesn't add it to routing table (the route never gets to be "best path", although it should be).
here's a config snippet:
!
ipv6 unicast-routing
!
01-16-2011 11:44 AM
Hello Jakob,
You have an interesting issue. Can you please answer a few more questions?
Thank you!
Best regards,
Peter
01-16-2011 03:53 PM
hi,
you're right, i forgot to add a few details ...
1) yes, ip routing is on and working fine
2) i'm using "desktop IPv4 and IPv6 routing" template
3) output of show ip bgp ipv6 unicast 2001:1638::/32:
BGP routing table entry for 2001:1638::/32, version 0
Paths: (1 available, no best path)
Not advertised to any peer
6939 51058
2001:65F:E1:311::2 (FE80::20C:42FF:FE54:EFA8) (inaccessible) from 2001:65F:E1:311::2 (10.0.0.2)
Origin IGP, localpref 100, valid, internal
4) i used debug bgp ipv6 unicast and debug bgp ipv6 unicast updates
OK, seems we're getting somewhere, as the link-local address is marked inaccessible (guess I missed that before!). OTOH, the link-local ip is pingable:
sw01#ping ipv6 FE80::20C:42FF:FE54:EFA8
Output Interface: vlan311
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::20C:42FF:FE54:EFA8, timeout is 2 seconds:
Packet sent with a source address of FE80::217:EFF:FE4C:4447
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/8 ms
... and here's
sw01#show ipv6 neighbors vlan 311
IPv6 Address Age Link-layer Addr State Interface
2001:65F:E1:311::2 0 000c.4254.efa8 REACH Vl311
FE80::20C:42FF:FE54:EFA8 0 000c.4254.efa8 REACH Vl311
seems like a link-local problem?
=> enabling debug ipv6 nd
log:
Jan 16 23:47:52: ICMPv6-ND: Received NS for FE80::217:EFF:FE4C:4447 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:48:22: ICMPv6-ND: Received RA from FE80::20C:42FF:FE54:EFA8 on Vlan311
Jan 16 23:48:30: ICMPv6-ND: REACH -> STALE: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:18: ICMPv6-ND: Received NS for 2001:65F:E1:311::1 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:18: ICMPv6-ND: STALE -> DELAY: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: DELAY -> PROBE: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: Sending NS for FE80::20C:42FF:FE54:EFA8 on Vlan311
Jan 16 23:50:23: ICMPv6-ND: Received NA for FE80::20C:42FF:FE54:EFA8 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:23: ICMPv6-ND: PROBE -> REACH: FE80::20C:42FF:FE54:EFA8
Jan 16 23:50:28: ICMPv6-ND: Received NS for FE80::217:EFF:FE4C:4447 on Vlan311 from FE80::20C:42FF:FE54:EFA8
Jan 16 23:51:06: ICMPv6-ND: REACH -> STALE: FE80::20C:42FF:FE54:EFA8
so, link-local ip seems to "flap" rather often.
thanks,
jakob
12-22-2011 01:41 AM
I also have this problem with iBGP ipv6 routes on catalyst 4506e (IOS XE 03.02.02.SG). Here is one such ignored route:
#show ip bgp ipv6 unicast 2A03:CA00:CA00:867::/64
BGP routing table entry for 2A03:CA00:CA00:867::/64, version 0
Paths: (1 available, no best path)
Not advertised to any peer
Local
2A03:CA00:922::1:0 (FE80::1E6F:65FF:FE38:7170) (inaccessible) from 2A03:CA00:922::1:0 (213.187.127.105)
Origin incomplete, localpref 100, valid, internal
2A03:CA00:922::1:0 _is_ accessible as shown below:
#show ipv6 route 2A03:CA00:922::1:0
Routing entry for 2A03:CA00:922::/64
Known via "connected", distance 0, metric 0, type connected
Redistributing via ospf 1
Backup from "ospf 1 [110]"
Route count is 1/1, share count 0
Routing paths:
directly connected via Vlan922
Last updated 20:55:47 ago
#ping 2A03:CA00:922::1:0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2A03:CA00:922::1:0, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/5/12 ms
01-16-2011 12:15 PM
There is insufficient info in your starting post to resolve the issue.
Please follow the troubleshooting steps in the document below.
If this doesn't resolve the problem, please provide the output as shown in the flowchart under the section:
regards,
Leo
01-16-2011 03:38 PM
hi,
I just did that, the flowchart is telling me to open a TAC case
here's how i reached that point:
path is in bgp table => learned from iBGP => no synchronization => next hop is valid and reachable => path isn't marked as received only (afaik)
note that this flowchart assumes a "best path". in my case, however, the path isn't getting marked "best" in the first place
01-16-2011 10:19 PM
I would not be too sure of that. Please verify your ipv6 neighbor states.
What I also noticed was the peculiar use of ipv6 addresses. You should use the eui64 convention as in the example below:
http://www.cisco.com/en/US/tech/tk872/technologies_configuration_example09186a0080b4a32f.shtml#diag
Using this convention maps the link-local address and any network address to the same host part of the address thus reducing the required number of multicast addresses. Without a lab, I cannot test it but the fact that this is not the case in your config may be a source of trouble.
This would make your route unreachable by an invalid next-hop. This was also the reason why I advised you to check ipv6 neigbor state.
regards,
Leo
01-17-2011 04:27 AM
thanks for your input!
EUI-64 (or more precisely, SLAAC) is by no means a convention or requirement! why should it be "peculiar" to statically configure an address? for routers and servers, where ip addresses hardly every change, i'd like to have everything statically and manually assigned, makes so life much easier. also, SLAAC doesn't offer any real advantage here.
i have other routers online (linux-based, juniper, etc.) which aren't configured SLAAC yet add bgp routes to the routing table properly :-)
as far as i can tell, ND does work. i'm seeing both (global and link-local) addresses in REACH state. i can ping both successfully. only thing i notice is a somewhat frequent flapping of neighbor states from REACH to STALE and back.
I have no idea why bgp believes the peer's FE80 is inaccessible ...
01-17-2011 06:25 AM
Hi,
FE80::20C:42FF:FE54:EFA8
this is a SLAAC address indeed but the the 7th bit in first octet of mac address hasn't been flipped so it is not a modified eui-64 but simply a eui-64 but nonetheless it isn't a static one
Regards.
alain.
01-17-2011 08:43 AM
Alain,
this is a SLAAC address indeed but the the 7th bit in first octet of mac address hasn't been flipped
Hmmm, are you sure about that? The modified EUI-64 here is 020C:42FF:FE54:EFA8. After removing the FF:FE word, we get 020C:4254:EFA8. The first byte is 0x02 having clearly the 7th bit set to 1, so it is flipped after all.
In any case, something very fishy is going on here with the BGP...
Best regards,
Peter
01-17-2011 12:14 PM
Hi Peter,
Hmmm, are you sure about that? The modified EUI-64 here is 020C:42FF:FE54:EFA8. After removing the FF:FE word, we get 020C:4254:EFA8. The first byte is 0x02 having clearly the 7th bit set to 1, so it is flipped after all.
You're right and I am ashamed having posted such a stupidity
Kind Regards.
Alain.
01-17-2011 12:48 PM
Hello Alain,
Don't even mention it. A few months ago, I posted a question about PIM Register messages stating that they are strangely addressed - to the multicast address instead of the RP address. I even developed a large hypothesis why that is so. Only two days later I've realized that the Wireshark was feeding me with the IP address of the innermost multicast IP packet but the outer IP packet with the PIM Register message was addressed completely correctly, to the RP. You can imagine my feelings at that moment
Best regards,
Peter
01-17-2011 09:18 AM
Hello Chris,
This just does not make sense. I have concocted a VERY quick scenario similar to your, and I have retaken most of your configs. I have connected my 3560V2, IOS 12.2(55)SE IPSERVICESK9, to a 1841 router running 15.0(1)M4 ADVENTERPRISEK9 (Fa0/23 on the 3560, Fa0/0 on the router). The relevant configs are:
3560:
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Switch
!
boot-start-marker
boot-end-marker
!
!
!
!
no aaa new-model
system mtu routing 1500
ip routing
!
!
ipv6 unicast-routing
!
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
interface FastEthernet0/1
!
interface FastEthernet0/2
!
interface FastEthernet0/3
!
interface FastEthernet0/4
!
interface FastEthernet0/5
!
interface FastEthernet0/6
!
interface FastEthernet0/7
!
interface FastEthernet0/8
!
interface FastEthernet0/9
!
interface FastEthernet0/10
!
interface FastEthernet0/11
!
interface FastEthernet0/12
!
interface FastEthernet0/13
!
interface FastEthernet0/14
!
interface FastEthernet0/15
!
interface FastEthernet0/16
!
interface FastEthernet0/17
!
interface FastEthernet0/18
!
interface FastEthernet0/19
!
interface FastEthernet0/20
!
interface FastEthernet0/21
!
interface FastEthernet0/22
!
interface FastEthernet0/23
switchport access vlan 311
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/24
!
interface GigabitEthernet0/1
!
interface GigabitEthernet0/2
!
interface Vlan1
no ip address
shutdown
!
interface Vlan311
ip address 10.0.0.1 255.255.255.224
no ip redirects
no ip unreachables
no ip proxy-arp
ipv6 address 2001:65F:E1:311::1/64
ipv6 enable
!
router bgp 12345
bgp router-id 10.9.1.1
bgp log-neighbor-changes
neighbor 2001:65F:E1:311::2 remote-as 12345
!
address-family ipv4
no neighbor 2001:65F:E1:311::2 activate
no auto-summary
no synchronization
exit-address-family
!
address-family ipv6
neighbor 2001:65F:E1:311::2 activate
network 2001:65F:E1::/48
no synchronization
exit-address-family
!
ip classless
ip http server
ip http secure-server
!
!
ip sla enable reaction-alerts
!
!
!
!
line con 0
line vty 5 15
!
end
1841:
version 15.0
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
!
!
!
dot11 syslog
ip source-route
!
!
!
!
ip cef
ipv6 unicast-routing
ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
license udi pid CISCO1841 sn FCZ140890CS
!
redundancy
!
!
controller DSL 0/1/0
!
!
!
!
!
!
!
!
interface Loopback10
no ip address
ipv6 address 2001:1638::1/32
!
!
interface FastEthernet0/0
no ip address
duplex auto
speed auto
ipv6 address 2001:65F:E1:311::2/64
!
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
!
interface Serial0/0/0
no ip address
shutdown
no fair-queue
clock rate 2000000
!
!
interface Serial0/0/1
no ip address
shutdown
clock rate 2000000
!
!
router bgp 12345
bgp router-id 10.9.2.2
bgp log-neighbor-changes
neighbor 2001:65F:E1:311::1 remote-as 12345
!
address-family ipv4
no synchronization
no neighbor 2001:65F:E1:311::1 activate
no auto-summary
exit-address-family
!
address-family ipv6
no synchronization
network 2001:1638::/32
neighbor 2001:65F:E1:311::1 activate
exit-address-family
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
!
!
!
!
!
!
!
control-plane
!
!
!
line con 0
line aux 0
line vty 0 4
login
!
scheduler allocate 20000 1000
end
I did not shorten the configs, just to be sure nothing is omitted - once again, nothing has been removed from these configs, they are posted as-are.
Now, on the 3560:
Switch#show ip bgp ipv6 unicast sum
BGP router identifier 10.9.1.1, local AS number 12345
BGP table version is 2, main routing table version 2
1 network entries using 141 bytes of memory
1 path entries using 76 bytes of memory
2/1 BGP path/bestpath attribute entries using 280 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 497 total bytes of memory
BGP activity 1/0 prefixes, 1/0 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2001:65F:E1:311::2
4 12345 21 17 2 0 0 00:15:26 1
Switch#show ip bgp ipv6 unicast
BGP table version is 2, local router ID is 10.9.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i2001:1638::/32 2001:65F:E1:311::2
0 100 0 i
Switch#show ip bgp ipv6 unicast 2001:1638::/32
BGP routing table entry for 2001:1638::/32, version 2
Paths: (1 available, best #1, table Default)
Not advertised to any peer
Local
2001:65F:E1:311::2 from 2001:65F:E1:311::2 (10.9.2.2)
Origin IGP, metric 0, localpref 100, valid, internal, best
Switch#show ipv6 route
IPv6 Routing Table - Default - 4 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
B - BGP, R - RIP, D - EIGRP, EX - EIGRP external
ND - Neighbor Discovery
O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
C 2001:65F:E1:311::/64 [0/0]
via Vlan311, directly connected
L 2001:65F:E1:311::1/128 [0/0]
via Vlan311, receive
B 2001:1638::/32 [200/0]
via 2001:65F:E1:311::2
L FF00::/8 [0/0]
via Null0, receive
Switch#ping 2001:1638::1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:1638::1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/1/8 ms
Switch#show ipv6 neigh
IPv6 Address Age Link-layer Addr State Interface
2001:65F:E1:311::2 0 ec44.7682.dc12 REACH Vl311
So, I have no problems in establishing the BGP routing and getting the route into the routing table. The question is, what have I done differently? Can you compare your configuration with mine very closely and try to discover the differences? (Note that I have given the router the BGP Router-ID 10.9.2.2, you have a different Router-ID - are you sure it is unique?)
Sorry for a lengthy post. And oh, by the way, are you able to test a different IOS version on your 3560?
Best regards,
Peter
01-17-2011 10:16 AM
Great work Peter! And intersting scenario.
OP has edited his 2nd posting with additional info, as you certainly will have seen. This shows the problem is very likely a broken ipv6 neigbor state.
This leads BGP to set the next-hop as invalid. As there was little info on the router side, it remains guesswork to the exact cause of this.
Two things I noticed are:
1: a reference to 10.0.0.2 in the unreachable message.
Where is 10.0.0.2? (Router side BGP router-id?) This may conflict with 10.0.0.1/28 on the 3560.
2: The ip access group out on vlan 311. (Can we presume this is an ipv4 acl?)
Could this perhaps affect ipv6 operation? At least it is one of the things missing in your testconfig.
We have no info on how the acl looks.
regards,
Leo
01-17-2011 11:20 AM
Peter, Leo,
once again, thanks for your input!
I don't notice any obvious difference from your test config to mine.
I'm sure my router-id is unique
I can't reboot this switch because it's in production use, unfortunately. But I agree, could be problem where a single reboot magically resolves everything
the ipv4 address in brackets in the route output is the (correct) ipv4 address of the bgp peer
I also temporarily removed the ipv4 access-group on vlan311, but didn't change a thing either.
my bgp state is fine, here's output from show bgp ipv6 unicast neighbor:
fsw01#show bgp ipv6 unic nei 2001:65F:E1:311::2
BGP neighbor is 2001:65F:E1:311::2, remote AS 12345, internal link
Description: rtr01
BGP version 4, remote router ID 10.0.0.2
BGP state = Established, up for 00:01:25
Last read 00:00:29, last write 00:00:25, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
Route refresh: advertised and received(new)
Address family IPv6 Unicast: advertised and received
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent Rcvd
Opens: 16 16
Notifications: 0 0
Updates: 17 19499
Keepalives: 5775 6623
Route Refresh: 0 1
Total: 5808 26139
Default minimum time between advertisement runs is 0 seconds
For address family: IPv6 Unicast
BGP table version 4, neighbor version 4/0
Output queue size : 0
Index 2, Offset 0, Mask 0x4
2 update-group member
Sent Rcvd
Prefix activity: ---- ----
Prefixes Current: 1 1 (Consumes 76 bytes)
Prefixes Total: 1 1
Implicit Withdraw: 0 0
Explicit Withdraw: 0 0
Used as bestpath: n/a 0
Used as multipath: n/a 0
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
Total: 0 0
Number of NLRIs in the update sent: max 1, min 1
Address tracking is disabled
Connections established 16; dropped 15
Last reset 00:01:44, due to User reset
Transport(tcp) path-mtu-discovery is enabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Mininum incoming TTL 0, Outgoing TTL 255
Local host: 2001:65F:E1:311::1, Local port: 14569
Foreign host: 2001:65F:E1:311::2, Foreign port: 179
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x56A6BEB7C):
Timer Starts Wakeups Next
Retrans 5 0 0x0
TimeWait 0 0 0x0
AckHold 5 3 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 1 0 0x56A734E49
DeadWait 0 0 0x0
iss: 1891638869 snduna: 1891639047 sndnxt: 1891639047 sndwnd: 5760
irs: 3937504546 rcvnxt: 3937504737 rcvwnd: 16194 delrcvwnd: 190
SRTT: 146 ms, RTTO: 1283 ms, RTV: 1137 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms
Flags: higher precedence, nagle, path mtu capable
Datagrams (max data segment is 1440 bytes):
Rcvd: 10 (out of order: 0), with data: 5, total data bytes: 190
Sent: 11 (retransmit: 0), with data: 11, total data bytes: 644
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide