Proposition
In order to gain scalable efficiency I proposed moving all DHCP physical network interfaces to a virtual network interface and bridging the virtual interface (BVI) back to a physical interface (l2 transport) logically. We would gain efficiency that is indeed scalable because we would be grouping OLTs logically based on what router they physically connect to, sharing IPv4 resources across multiple OLTs rather then dedicating IPv4 blocks to a singular OLT. I projected we would gain approximately 34% IP allocation efficiency from 54% to and overall efficiency of 89% across the network of 4 ASR's and 15 OLT's. This change would not only improve the allocation efficiency it would free up around 4078 IPs for reallocation. That means we can continue offering services as we currently do for at least 2 more years, before we need to talk NAT or more IPv4 space.
Case Study Deployment Overview and Results
I wanted to share a creative use of BVI (Bridged Virtual Interface) on the Cisco ASR platform to over come some IP allocation and utilization hurdles you might face as a service provider. We selected one market to test this deployment, to validate the scalability and performance. A little about me, I'm a network engineer for Elevate, a fiber to the home internet provider. We utilize the Cisco 9006 ASR as our layer 3 routing platform. We use Calix E720 OLT's as our layer 2 GPON switching platform to deliver fiber to the home in combination with the Calix 844g gigga centers. We serve a very rural population and with that our network today is very linear, we have a layer 3 ring but when it comes to the OLTs its very much hub and spoke not a lot of opportunity for rings or to make use of ERPS.
A little about GPON (gigabit passive optical network), each GPON connection has a 2.5Gbps shared capacity, and ONT although it has the GPON connection for its WAN, it still hands off to an ethernet 1Gbps copper connection, that can be delivered as a bridge for BYOR (bring your own router), or a router gateway on the giggacenter ONT. That leaves 1.5Gbps of capacity headroom. In a recent case study we did on our membership most users still only use 30Mbps to 50Mbps. We deliver 2 symmetric speed packages, 150Mbps and 1Gbps.
Our problem is that we serve 3 to 6 OLTs off of an ASR. Prior we had built and managed our DHCP networks using larger blocks such as /23's or even /21's and as the need arose we stack /24's on top of that. That's great but we wind up with 50% IP utilization on a /23 on some OLTs and 89% utilization on others. The argument can be made that we should have just broken the IP segments into smaller blocks such as /24s and stacked those. That is true in some cases but to maximize subnetting its remains more efficient to deploy larger networks if the projected utilization forecast calls for it. The issue with that is when the network is young and still being built out, it doesn't scale. By that I mean your stuck with 40% to 60% utilization while that market grows. This deployment strategy locks you into not having flexibility as you deploy more markets in parallel with the finite number of IPv4 you have to work with. In a perfect world everyone would use IPv6 but that's still not the case yet, So we got creative.
By using a BVI to bridge (3 - 6) OLT's (in terms of their DHCP networks) which for us is 90% residential and the great majority of the deployment, we where able to successfully and efficiently scale the network as we grow without locking us into dedicated allocation for most or all of our IPv4 space we current have. By doing this we where able to serve the same amount of customers (2307) with 28% less IP addresses allocated by sharing the networks. So the question is, how?
Here is a before configuration example.
ipv4 address 1.206.72.1 255.255.254.0 will be pulled back
ipv4 address 1.206.90.1 255.255.254.0 will be pulled back
ipv4 address 1.206.93.1 255.255.255.0 will be pulled back
ipv6 address 2001:230:a:9::1/64 will be pulled back
ipv6 address 2001:230:a:b::1/64 will be pulled back
ipv6 address 2001:230:a:d::1/64 will be pulled back
ipv6 address 2001:230:a:c::1/64 will be pulled back
ipv6 address 2001:230:a:e::1/64 will be pulled back
!
no interface Bundle-Ether4.700
description DHCP: Orchard City DHCP ONT
ipv4 address 1.206.72.1 255.255.254.0
ipv4 address 1.206.90.1 255.255.254.0 secondary
ipv4 address 1.206.93.1 255.255.255.0 secondary
ipv6 address 2001:230:a:9::1/64
ipv6 enable
encapsulation dot1q 700
!
no interface Bundle-Ether5.800
description DHCP: Hotchkiss DHCP ONT
ipv4 address 1.206.84.1 255.255.254.0
ipv4 address 1.206.67.1 255.255.255.0 secondary
ipv6 address 2001:230:a:a::1/64
ipv6 enable
encapsulation dot1q 800
!
no interface Bundle-Ether6.900
description DHCP: Cedaredge 1 DHCP ONT
ipv4 address 1.206.86.1 255.255.254.0
ipv4 address 1.206.74.1 255.255.254.0 secondary
ipv6 address 2001:230:a:b::1/64
ipv6 enable
encapsulation dot1q 900
!
no interface Bundle-Ether8.600
description DHCP: Delta CNL DHCP ONT
ipv4 address 1.206.89.129 255.255.255.128
ipv6 address 2001:230:a:d::1/64
ipv6 enable
encapsulation dot1q 600
!
no interface Bundle-Ether9.900
description DHCP: Cedaredge 2 DHCP ONT
ipv4 address 1.206.92.1 255.255.255.0
ipv4 address 1.10.73.1 255.255.255.0 secondary
ipv6 address 2001:230:a:c::1/64
ipv6 enable
encapsulation dot1q 900
!
no interface Bundle-Ether10.700
description DHCP: Garenet Mesa DHCP ONT
ipv4 address 1.10.85.1 255.255.255.0
ipv6 address 2001:230:a:e::1/64
ipv6 enable
encapsulation dot1q 700
!
no dhcp ipv4 profile CED-RES-DHCP relay
no dhcp ipv4 profile DLT-ONT-DHCP relay
no dhcp ipv4 profile HTC-RES-DHCP relay
no dhcp ipv4 profile ORC-RES-DHCP relay
no dhcp ipv4 profile CED-02-RES-DHCP relay
no dhcp ipv4 profile GNM-01-ONT-DHCP relay
Here is an after configuration example.
interface Bundle-Ether4.700 l2transport
description DHCP: Orchard City DHCP ONT
encapsulation dot1q 700
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface Bundle-Ether5.800 l2transport
description DHCP: Hotchkiss DHCP ONT
encapsulation dot1q 800
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface Bundle-Ether6.900 l2transport
description DHCP: Cedaredge 1 DHCP ONT
encapsulation dot1q 900
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface Bundle-Ether8.600 l2transport
description DHCP: Delta CNL DHCP ONT
encapsulation dot1q 600
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface Bundle-Ether9.900 l2transport
description DHCP: Cedaredge 2 DHCP ONT
encapsulation dot1q 900
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface Bundle-Ether10.700 l2transport
description DHCP: Garenet Mesa DHCP ONT
encapsulation dot1q 700
rewrite ingress tag pop 1 symmetric
l2protocol cpsv reverse-tunnel
!
interface bvi 600
description DHCP: ONT DHCP INTERNET
bandwidth 4294967295
mtu 9216
ipv4 address 1.206.84.1 255.255.254.0
ipv4 address 1.206.67.1 255.255.255.0 secondary
ipv4 address 1.206.86.1 255.255.254.0 secondary
ipv4 address 1.206.74.1 255.255.254.0 secondary
ipv4 address 1.206.89.129 255.255.255.128 secondary
ipv4 address 1.206.92.1 255.255.255.0 secondary
ipv4 address 1.10.73.1 255.255.255.0 secondary
ipv4 address 1.10.85.1 255.255.255.0 secondary
ipv6 nd ra-interval 60
ipv6 address 2001:230:a:a::1/64
ipv6 enable
!
!
dhcp ipv4
profile DLT-ASR-DHCP relay
helper-address vrf mgmt 172.20.99.46 giaddr 1.206.84.1
helper-address vrf mgmt 172.20.100.12 giaddr 1.206.84.1
relay information option
relay information option allow-untrusted
!
interface BVI600 relay profile DLT-ASR-DHCP
!
!
dhcp ipv6
interface BVI600 proxy profile DHCP-IPV6
!
!
l2vpn
bridge group DLT-ASR-DHCP
bridge-domain DLT-ASR-DHCP
mac
limit
maximum 128000
interface Bundle-Ether 4.700
!
interface Bundle-Ether 5.800
!
interface Bundle-Ether 6.900
!
interface Bundle-Ether 8.600
!
interface Bundle-Ether 9.900
!
interface Bundle-Ether 10.700
!
routed interface BVI600
!
!
Deployment observations
After deploying this configuration we noticed the broadcast traffic increased on the bridged broadcast domain to about 18Mbps for almost 3000 subscribers. For us, this is not an issue because of the amount of headroom we have in the technologies we chose to use. We did not see any performance degradation. We ran speed tests using our on net speedtest server and iperf as well as MTR's to ensure not packet loss, jitter or latency was introduced by the change. We where able to reclaim about 1,271 IPv4 address's in the form of 2 - /23's and 1 - /24 in this market. This equates to $27,962 at $22/IP if we where to broker a block today. This configuration buys us the time we need to continue deployment of new services at roughly 75 new customers per week. In addition to budgeting for additional IPv4 space or budgeting for a NAT64 and or CGN appliance(s). Before we redeployed IPs using this BVI strategy, we would have had to spend any where from $120,000 to $500,000 to deploy a NAT solution that is carrier grade and can scale or purchase more IPv4 space. For a growing ISP network, this deployment strategy just delays the inevitable, however if the bottom line is a factor in your engineering department this could be a viable solution for you, that allows you to gain flexibility.
Systems and Protocols used
DHCP Shared Networks (Vendor - Efficient IP)
Gateway (Vendor - Cisco)
OLT (Vendor - Calix)
ONT (Vendor - Calix)
VRF
IPv4
IPv6
DHCP Relay
DHCP Proxy
GPON
BVI
L2 transport
L2VPN
Bridge Group
Bridge Domain
