IPV6 Swtiching / Routing problem

[JasonTurnbull9467]

Hello,

I'm new to the community and I'm not sure if I'm asking in the right place. I'm also a total amateur but I've almost got the setup I need completed but now I'm stumped. Any help would be great.

I've got a Cisco router and switch. On the router, I have a tunnel out to an IPV6 tunnel broker service (This is working) I've got some bgp going on on my router as well. All of this for my own /32 of IPV6 addresses. I've got a trunked port out of my switch into my router, and a trunked port out of my switch off to a local host for vlan purposes.

I've got sub-interfaces on the router with the proper encapsulation and ID's set. I've got the vlans I need created on the swtich.

Over on my host machine, I try to connect out to the internet via my vlans and have no luck. I can utilize the block of address space I assigned to vlan 1 on the switch. Since vlan 1 is native all of that traffic heads over my hosts main interface. Say eth0. I can setup eth.1 , eth.2, etc and ping the switch address assigned to these vlans, as well as the gateway addresses assigned to the sub-interfaces on the router. But if I try to ping from one of the Vlans to something other than it's own subnet, I get nothing.

I'm not sure If I've explained it well enough...but I've been struggling for awhile and figured I'd ask. In case someone knows what I may be missing. 

Thanks in advance! 

[JasonTurnbull9467]

Thank you for the help Harold. I appreciate it.

[Harold Ritter]

Hi Jason,

You are welcome. By the way, I ran a quick test and got it to work without VRFs, but I had to change the /35 for /64 on the router.

here's the output from the server with the various default gateway all installed in the main RIB.

cisco@vpp-2:~$ netstat -rn -6
Kernel IPv6 routing table
Destination Next Hop Flag Met Ref Use If
2602:fec2::/64 :: U 256 0 0 eth1
2602:fec2::/35 :: UAe 256 0 0 eth1
2602:fec2:2000::/64 :: U 256 0 0 eth1.100
2602:fec2:4000::/64 :: UAe 256 0 0 eth1.200
2602:fec2:6000::/64 :: U 256 0 0 eth1.300
2602:fec2:6000::/64 :: UAe 256 0 0 eth1
fe80::/64 :: U 256 0 0 eth1
fe80::/64 :: U 256 0 0 eth1.100
fe80::/64 :: U 256 0 0 eth1.200
fe80::/64 :: U 256 0 0 eth1.300
::/0 fe80::f816:3eff:fe01:930e UGDAe 1024 0 0 eth1
::/0 fe80::f816:3eff:fe01:930e UGDAe 1024 0 0 eth1.100
::/0 fe80::f816:3eff:fe01:930e UGDAe 1024 0 0 eth1.200
::/0 fe80::f816:3eff:fe01:930e UGDAe 1024 0 0 eth1.300
::/0 :: !n -1 1 250 lo
::1/128 :: Un 0 1 4 lo
2602:fec2::3/128 :: Un 0 1 23 lo
2602:fec2::f816:3eff:fe5c:bb96/128 :: Un 0 1 18 lo
2602:fec2:2000::3/128 :: Un 0 1 6 lo
2602:fec2:2000:0:f816:3eff:fe5c:bb96/128 :: Un 0 1 3 lo
2602:fec2:4000::3/128 :: Un 0 1 1 lo
2602:fec2:4000:0:f816:3eff:fe5c:bb96/128 :: Un 0 1 0 lo
2602:fec2:6000::3/128 :: Un 0 1 1 lo
2602:fec2:6000:0:f816:3eff:fe5c:bb96/128 :: Un 0 1 0 lo
2602:fec2:6000:0:f816:3eff:fe5c:bb96/128 :: Un 0 1 26 lo
fe80::f816:3eff:fe5c:bb96/128 :: Un 0 1 13 lo
fe80::f816:3eff:fe5c:bb96/128 :: Un 0 1 5 lo
fe80::f816:3eff:fe5c:bb96/128 :: Un 0 1 2 lo
fe80::f816:3eff:fe5c:bb96/128 :: Un 0 1 10 lo
ff00::/8 :: U 256 0 0 eth1
ff00::/8 :: U 256 1 0 eth1.100
ff00::/8 :: U 256 1 0 eth1.200
ff00::/8 :: U 256 0 0 eth1.300
::/0 :: !n -1 1 250 lo

And pinging to destination beyond the router works as well for all prefixes.

Regards,

[JasonTurnbull9467]

So on the router, set the sub-interfaces to /64's instead of /35's? Do you have a set of configs, router and/or switch from the test you did that I could look over? Either way, this is great news. I was working on the vrf based solution and having trouble. May have been from the masking.

Also, I'm curious about the fe80::e23f:49ff:feb1:a81e/128 addresses in your netstat. They appear to be link-local and configured as individual gateaways per vlan? Where is that address coming from?

EDIT: I think that is the link-local address for the router? When I manually add routes to my table on the host for each vlan to my router interfaces link-local everything works! I'd still love to see your test router and switch configs but I think I may have it working thanks to you.

[Harold Ritter]

> So on the router, set the sub-interfaces to /64's instead of /35's? Do you have a set of configs, router and/or switch from the > test you did that I could look over?

Yes, that is correct. Here is the config excerpt from the router.

interface GigabitEthernet0/1
ipv6 address 2602:FEC2::1/64
!
interface GigabitEthernet0/1.100
encapsulation dot1Q 100
ipv6 address 2602:FEC2:2000::1/64
!
interface GigabitEthernet0/1.200
encapsulation dot1Q 200
ipv6 address 2602:FEC2:4000::1/64
!
interface GigabitEthernet0/1.300
encapsulation dot1Q 300
ipv6 address 2602:FEC2:6000::1/64
!

> I think that is the link-local address for the router? When I manually add routes to my table on the host for each vlan to my

> router interfaces link-local everything works! 

Yes, the default route installed in the server RIB come from the received router advertisement received on each sub-interface. You will not need to configure them manually, if you change the /35 to /64 on the router side.

Regards,

[JasonTurnbull9467]

I got it working but I did have to manually add all of the routes. I only get a single default route and its not the link local address for the router....is it possible I have advertisements configured wrong? Could be a limitation in Ubuntu Servers networking.

[Harold Ritter]

I am using Ubuntu as the server as well. You should be receiving a router advertisement (RA) per subinterface and the server should install, unless the accept_ra sysctl parameter has been disabled.

cisco@vpp-2:~$ uname -a
Linux vpp-2 3.13.0-52-generic #85-Ubuntu SMP Wed Apr 29 16:44:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
cisco@vpp-2:~$ sudo tcpdump -i eth1.100 -s 1500 -vv icmp6
tcpdump: WARNING: eth1.100: no IPv4 address assigned
tcpdump: listening on eth1.100, link-type EN10MB (Ethernet), capture size 1500 bytes
18:18:31.543798 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::f816:3eff:fe12:d593 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 64
hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
source link-address option (1), length 8 (1): fa:16:3e:12:d5:93
0x0000: fa16 3e12 d593
mtu option (5), length 8 (1): 1500
0x0000: 0000 0000 05dc
prefix info option (3), length 32 (4): 2602:fec2:2000::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
0x0000: 40c0 0027 8d00 0009 3a80 0000 0000 2602
0x0010: fec2 2000 0000 0000 0000 0000 0000
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
cisco@vpp-2:~$ sudo tcpdump -i eth1.200 -s 1500 -vv icmp6
tcpdump: WARNING: eth1.200: no IPv4 address assigned
tcpdump: listening on eth1.200, link-type EN10MB (Ethernet), capture size 1500 bytes
18:21:27.983803 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::f816:3eff:fe12:d593 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 64
hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
source link-address option (1), length 8 (1): fa:16:3e:12:d5:93
0x0000: fa16 3e12 d593
mtu option (5), length 8 (1): 1500
0x0000: 0000 0000 05dc
prefix info option (3), length 32 (4): 2602:fec2:4000::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
0x0000: 40c0 0027 8d00 0009 3a80 0000 0000 2602
0x0010: fec2 4000 0000 0000 0000 0000 0000
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
cisco@vpp-2:~$ sudo tcpdump -i eth1.300 -s 1500 -vv icmp6
tcpdump: WARNING: eth1.300: no IPv4 address assigned
tcpdump: listening on eth1.300, link-type EN10MB (Ethernet), capture size 1500 bytes
18:23:15.304926 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::f816:3eff:fe12:d593 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 64
hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
source link-address option (1), length 8 (1): fa:16:3e:12:d5:93
0x0000: fa16 3e12 d593
mtu option (5), length 8 (1): 1500
0x0000: 0000 0000 05dc
prefix info option (3), length 32 (4): 2602:fec2:6000::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
0x0000: 40c0 0027 8d00 0009 3a80 0000 0000 2602
0x0010: fec2 6000 0000 0000 0000 0000 0000
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
cisco@vpp-2:~$

cisco@vpp-2:~$ sudo sysctl -a | grep "accept_ra "
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.eth0.accept_ra = 1
net.ipv6.conf.eth1.accept_ra = 1
net.ipv6.conf.eth1/100.accept_ra = 1
net.ipv6.conf.eth1/200.accept_ra = 1
net.ipv6.conf.eth1/300.accept_ra = 1
net.ipv6.conf.lo.accept_ra = 1
cisco@vpp-2:~$

Regards,

[JasonTurnbull9467]

All up and working fine now. For some reason all of the vlan are set to accept_ra=0. I tried changing sysctl.conf and then I tried manually changing each individual vlan via sysctl -w net.ipv6.conf.enp5s0/2.accept_ra=1 but the change won't survive a reboot. I even threw all of the commands into a sysctl.d\local.conf file and they still won't survive a reboot. I've got an /etc/rc.local file adding all of the routes for me manually now and that's working. I have no idea whats overriding my sysctl settings. I'll look into it. Thanks again for all of your help Harold. I never would have got it working on my own.

[Harold Ritter]

Hi Jason,

I am glad it is now working for you.

≥ I even threw all of the commands into a sysctl.d\local.conf file and they still won't survive a reboot.

Did you check to see if there wasn't already a file in /etc/sysctl.d/ or /etc/sysctl.conf that changes that value? Files in that directory are normally prefixed with a numerical value (i.e. 10-local.conf) and the higher values are executed after, overriding the value in the lower value file. It might be a good idea to name your file 99-local.conf or something similar. Bear in mind that any value configured in /etc/sysctl.con will override any other value configured in files in /etc/sysctl.d/.

cisco@vpp-2:/etc/sysctl.d$ cat README
This directory contains settings similar to those found in /etc/sysctl.conf.
In general, files in the 10-*.conf range come from the procps package and
serve as system defaults. Other packages install their files in the
30-*.conf range, to override system defaults. End-users can use 60-*.conf
and above, or use /etc/sysctl.conf directly, which overrides anything in
this directory.

After making any changes, please run "service procps start" (or, from
a Debian package maintainer script "invoke-rc.d procps start").
cisco@vpp-2:/etc/sysctl.d$

Regards,

[JasonTurnbull9467]

Sorry for the late response. Some personal stuff came up. I managed to get it all working by adding accept_ra 1 to each interfaace block in my /etc/network/interfaces file. Apparently it overrides everything else. The other methods are probably not working due to something I messed with in an earlier attempt to bring up the vlans. Once I'm on a fresh server install I'm sure that method will work as expected.

In the mean time I'm dealing with a different issue. IPv4 related.

I have a sub-interface on g0/0/0 (my internet facing interface) as follows:

interface GigabitEthernet0/0/0.1
encapsulation dot1Q 2
ip address 216.115.150.34 255.255.255.192
ip nat outside
!

I have a static route on the router as follows:

ip nat inside source static 10.0.0.24 216.115.150.34

On the switch I have a dhcp pool : 

ip dhcp pool PROXYSERVER
host 10.0.0.24 255.255.255.0
hardware-address e03f.49b1.a81e

that pool hands that specific address to that host so that I can route it's traffic out over a seperate IP,  216.115.150.34

I try to ping 8.8.8.8 as a test from the host and get nothing. No errors just a hang. Through trial and error, I found that if I add a static route on the router like this:

ip route 10.0.0.24 255.255.255.255 GigabitEthernet0/0/0.1

and then REMOVE the route:

no ip route 10.0.0.24 255.255.255.255 GigabitEthernet0/0/0.1

I get connectivity. Adding the route alone isn't enough. I have to add it, and then remove it and then it works for a few hours. When I come back to work in the morning though, connectivity is gone. If I re-add the route and then remove it, it comes alive again.  No idea whats going on there.

EDIT: Actually, it's not even working when I add that route and remove it anymore. That worked 3 or 4 times but now, nothing. Didn't make sense why it worked in the first place so I'm not surprised.

[Harold Ritter]

Hi Jason,

I would personally start with disabling routing on the switch ("no ip routing") and move the DHCP configuration on the router.

Regards,

[JasonTurnbull9467]

I've disabled routing on the switch and moved both of my dhcp pools over to the router. So far I'm getting the same strange behavior. I'll keep at it. I assume it's something simple I'm overlooking.

[Harold Ritter]

Can you please post the relevant config from the router?

Regards,

[JasonTurnbull9467]

Sure,

Here it is:

Building configuration...

Current configuration : 5979 bytes
!
! Last configuration change at 18:40:27 UTC Wed Oct 9 2019
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname Cerberus
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable secret 5 xxxxxxxxxxxx
enable password xxxxxxxxxx
!
no aaa new-model
!
!
!
!
!
!
!
!
!

no ip domain lookup

no ip dhcp conflict logging
ip dhcp excluded-address 10.0.0.1
!
ip dhcp pool LanPool0.1
network 11.0.0.0 255.255.252.0
default-router 11.0.0.1
domain-name Cerberus.Local.0.1
dns-server 8.8.8.8 8.8.4.4
!
ip dhcp pool LanPool
network 10.0.0.0 255.255.255.0
default-router 10.0.0.1
domain-name Cerberus.Local
dns-server 8.8.8.8 8.8.4.4
!
ip dhcp pool PROXYSERVER
host 10.0.0.24 255.255.255.0
hardware-address e03f.49b1.a81e
!
!
!
ipv6 unicast-routing
ipv6 dhcp pool Proxy1001
dns-server 2001:4860:4860::8888
domain-name cerberus.local
!
ipv6 dhcp pool 32Block
dns-server 2001:4860:4860::8888
domain-name Cerberus
!
!
!
!
!
!
!
!
subscriber templating
multilink bundle-name authenticated
!
!
!

license udi pid ISR4331/K9 sn FDO19321C50
spanning-tree extend system-id
!
!
redundancy
mode none
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface Loopback1
ip address 192.59.31.1 255.255.255.0
!
interface Tunnel0
description Hurricane Electric IPv6 Tunnel Broker
no ip address
ipv6 address 2001:472:1F0C:BF::2/64
ipv6 enable
tunnel source 216.115.150.175
tunnel mode ipv6ip
tunnel destination 209.51.163.34
!
interface GigabitEthernet0/0/0
ip address 216.115.150.175 255.255.255.192
ip nat outside
negotiation auto
ipv6 enable
!
interface GigabitEthernet0/0/0.1
encapsulation dot1Q 2
ip address 216.115.150.34 255.255.255.192
!
interface GigabitEthernet0/0/1
ip address 10.0.0.1 255.255.255.0
ip nat inside
negotiation auto
ipv6 address 2602:fec2::1/35
ipv6 enable
ipv6 traffic-filter fromswitch in
ipv6 traffic-filter fromswitch out
!
interface GigabitEthernet0/0/1.2
encapsulation dot1Q 2
ipv6 address 2602:fec2:2000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.3
encapsulation dot1Q 3
ipv6 address 2602:fec2:4000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.4
encapsulation dot1Q 4
ipv6 address 2602:fec2:6000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.5
encapsulation dot1Q 5
ipv6 address 2602:fec2:8000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.6
encapsulation dot1Q 6
ipv6 address 2602:fec2:A000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.7
encapsulation dot1Q 7
ipv6 address 2602:fec2:C000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/1.8
encapsulation dot1Q 8
ipv6 address 2602:fec2:E000::1/35
ipv6 enable
!
interface GigabitEthernet0/0/2
ip address 142.176.184.242 255.255.255.252
negotiation auto
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
!
interface Vlan1
no ip address
!
router bgp 397759
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 2001:472:1F0C:BF::1 remote-as 6939
neighbor 2001:472:1F0C:BF::1 update-source Tunnel0
!
address-family ipv4
exit-address-family
!
address-family ipv6
network 2602:fec2::/32
neighbor 2001:472:1F0C:BF::1 activate
exit-address-family
!
ip default-gateway 216.115.150.129
ip nat inside source list 1 interface GigabitEthernet0/0/0 overload
ip nat inside source static 10.0.0.24 216.115.150.34 extendable
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 216.115.150.129
!
!
ip access-list extended BaseACL
permit icmp any any
ip access-list extended Manage-SSH
ip access-list extended NOSPOOF
permit icmp any any
permit ip any any
!
access-list 1 permit any
access-list 1 permit 10.0.0.0 0.0.0.255
access-list 1 permit 10.0.0.0 0.0.255.255
access-list 101 permit icmp any any
ipv6 route 2602:fec2::/32 Tunnel0
ipv6 route ::/0 Tunnel0
!
snmp-server community public RO
!
!
ipv6 access-list fromswitch
permit ipv6 any any
!
control-plane
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class 1 in
password xxxxxxxx
login
line vty 5 15
access-class 1 in
login
!
!
end

[Harold Ritter]

Do you have a route back from the Internet router to 216.115.150.34 pointing to 216.115.150.175. Why did you configure 216.115.150.34 on interface GigabitEthernet0/0/0.1? Where does this subinterface lead to? Can you explain what you are trying to do?

Regards,

[JasonTurnbull9467]

Again, if it's confusing I apologize. I'm still trying to get the hang of this. I have configured the sub-interface and setup a static nat route (ip nat inside source static 10.0.0.24 216.115.150.34 extendable) so that I can have that second public ip assigned to one host, 10.0.0.24 and have it reachable from the internet via the 216.115.150.34 address.

My ISP didn't give me a full block of ipv4 addresses to subnet but handed me two separate ones as far as I can tell. I need them both facing the internet and I need the .34 address to be assigned to the 10.0.0.24 machine 1 to 1. What you see in the config is my (partially working at least) attempt at this. I can hit the net via that .34 address on the 10.0.0.24 machine.

If theres a less convoluted way to accomplish what I'm asking, I'm all for it.

I'd assign the .34 address to the host directly but I can't figure out how to route it out to the net.

Thanks again for the help/