on 07-14-2013 05:11 AM
This document provides a guide on how to use the satellite (ASR9000v) with the ASR9000 and ASR9900 series routers. It will be discussed what you can and cannot do, how to verify the satellite operation and use cases.
This document is written assuming that 5.1.1 or greater software release will be used.
Satellite is a relatively new technology that was introduced in XR 4.2.1. Satellite provides you a great and cheap way to extend your 1G ports by using this port-extender which is completely managed out of the ASR9000. The advantage is that you may have 2 devices, but 1 single entity to manage. All the satellite ports are showing up in the XR configuration of the ASR9000.
Another great advantage of the Satellite is that you can put it on remote locations, miles away from the ASR9000 host!
Although there is a limit to the number of satellites you can connect to an ASR9000 (cluster), the Satellite general concept of ASR9000 is shown here in this picture:
The physical connections are very flexible. The link between the Satellite and the ASR9000 is called the ICL or "Inter Chassis Link".
This link transports the traffic from the satellite ports to the 9000 host.
In the ASR9000 host configuration you define how the satellite ports are mapping between the ICL and the satellite ports.
You can statically pin ports from the Satellite to a single uplink (that means there is no protection, when that uplink fails, those satellite
ports become unusable), or you can bundle uplinks together and assign a group of ports to that bundle. This provides redundancy,
but with the limitation that the satellite ports that are using an uplink bundle, can't be made part of a bundle themselves.
We'll talk about restrictions a bit later.
In the picture below you see Access Device A1 connecting with a bundle that uses uplink (green) to the 9k host LC-2.
A second satellite has all their ports in a bundle ICL.
Note that there is no bandwidth constraints enforced, so theoretically you can have a 2 member ICL bundle and 30 Satellite ports mapped to it, but that would mean there is oversubscription.
While the ASR9000v/Satellite is based on the Cisco CPT-50, you cannot convert between the 2 devices by loading different software images.
You can't use the 9000v as a standalone switch, it needs the ASR9000 host.
Visual differences include that the 9000v starts the port number at 0, where the CPT starts at 1. Also the CPT has 4 different power options
and the ASR9000v only 3: AC-A, DC-A, DC-E (A for Ansi, E for ETSI).
Satellite packet format over L1 topologies looks like this; there is a simple sneaky dot1q tag added which we call the nV tag:
In L2 topologies, such as simple ring, we use dot1ad.
There is a license required to run the ASR9000v. There are 3 licenses for 1, 5, or 20 Satellites per 9k host named:
While licenses are not hard enforced, this meaning the system will still work even though a license may not be present, however you are urged to obtain the proper license, syslog messages will show the "violation of use".
Note when using simple ring, a host license for each satellite is needed on each host. E.g. a simple ring with three satellites requires six A9K-NVSAT1-LIC licenses.
A variety of optics are supported on the ASR9000v, they may not be always the same as the ASR9000. Reference this link for the supported optics for ASR9000/9000v.
When using Tunable optics for the 9000v, pay attention to the following:
(*) note for the tunable optic on the IRL you need to set the wavelength the first time via the 9000v shell on insertion of the optic before shipping it to the destined location.
Handling of Unsupported Optics
For the 9000v ports we do not support the 'service unsupported-transceiver' or 'transceiver permit pid all' commands.
The satellite device simply flags an unsupported transceiver without disabling the port or taking any further action. As long as the pluggable is readable by the satellite the SFP may work, but there are no additional 'hacks' such as the hidden commands beyond what is shown as supported in the tables from the supported optic reference link.
The following software and hardware requirements exist for the ASR9000v. Although support started in XR4.2.1 My personal recommendation is to go with XR43 (latest) as many initial restrictions are lifted from the first release:
Minimum version is XR 4.2.1
Note: If the wrong port is used for ICL then the link will stay down on the 901. Once the correct ICL port is used and the 9K configured then a reload of the 901 will need to occur for the link to come up and the 901 become recognized as a satellite.
Generally speaking all features supported on physical GigE ports of ASR9K are also automatically supported on the GigE ports of the
satellite switch such as L2, L3, multicast, MPLS, BVI, OAM … (everything that works on a normal GigE).
–L1 features: applied on the satellite chassis
1) The following features are not supported on satellite ports in 5.1.1
*Need to update this*
2) When the ICL is a link bundle, there are some restrictions :
QoS can be applied on the ASR9000 host (runs on the NP where the satellite ports have their interface descriptors) or offloaded to the satellite
When you have oversubscription, that is more then the number of 1G ports compared to the ICL total speed there could be a potential issue. However there is an implicit trust model for all high priority traffic.
Automatic packet classification rules determine whether a packet is control packet (LACP, STP, CDP, CFM, ARP, OSPF etc), high priority data (VLAN COS 5,6,7, IP prec 5, 6, 7) or normal priority data and queued accordingly
For the downstream direction, that is 9000 host to the Satellite, the "standard" QOS rules and shaping are sufficient enough to warrant the delivery of high priority packets to the satellite ports. (e.g. inject to wire etc).
As the ICL link between the satellite and host may be oversubscribed by access interfaces, configuring QoS on the satellite itself is optimal for avoiding the lose of high priority traffic due to congestion. This feature was introduced in 5.1.1
3 steps to configuring QoS offload
INPUT access interface (CLI config) example:
class-map match-any my_class
match dscp 10
end-class-map
!
policy-map my_policy
class my_class
set precedence 1
!
end-policy-map
!
interface GigabitEthernet100/0/0/9
ipv4 address 10.1.1.1 255.255.255.0
nv
service-policy input my_policy
!
Traffic is hashed across members of this ICL LAG based on satellite access port number. No packet payload information (MAC SA/ DA or IP SA/ DA) used in hash calculation. This ensures that QoS applied on ASR9K for a particular satellite access port works correctly over the entire packet stream of that access port. Current hash rule is simple (port number modulo # of active ICLs)
Plug-and-play installation: No local config on satellite, no need to even telnet in!
It is recommended to use the auto-IP feature, no loopback or VRF need to be defined. A VRF **nVSatellite will be auto-defined and does not count towards the number of VRFs configured (for licensing purposes).
Optional config secret password for satellite login. Note that the username is 'root'
There are two options for ICL:
That is static pinning; designate some ports from the satellite to use a dedicated uplink.
Using a bundle ICL that provides for redundancy when one uplink fails.
All interfaces mapped to an ICL bundle:
ASR9000 TenG interface putting into bundle mode ON (No LACP support)
Define the bundle ethernet on the ASR9000 host, and designate which ports will be mapped to the bundle:
Because of the order and batching in which things get applied in XR there are some things that you need to know when it comes down to negating certain config which additions of others.
Examples of this are:
In such cases, failures are expected to be seen; generally speaking, failures are expected to be deterministic, and workarounds available
(re-apply the configuration in two commit batches)
Recommendation to users is to commit ICL configuration changes in separate commits to Satellite-Ether configuration changes
Starting in 5.1.1 many new features were added to expand upon the basic single host hub-and-spoke model. These features take more configuration than the base satellite configuration and will be discussed below.
Starting in 5.1.1 the ability for a satellite (hub-and-spoke) or a ring of satellites (simple ring) to be dual-homed to two hosts was added. (nV Edge acts as one logical router)
With this configuration one ASR9K host is the active and the other is standby. Data and control traffic from the satellite will flow to the active host, but both hosts will send and receive management traffic via the SDAC protocol. This is used to determine connectivity, detect failures, sync the configuration, etc.
The two hosts communicate management information via ICCP with a modified version of SDAC called ORBIT.
Supported Topologies:
Hub-and-spoke dual hosts
9000v with 10G ICL or bundle ICL
901 with 1G ICL
9000v (10G) or 901 (1G) using L2 fabric sub-interfaces
Satellites may be partitioned
Simple ring dual hosts
9000v with 10G ICL
901 with 1G ICL
Satellites may not be partitioned
Note: Partitioning is when you carve out certain access ports to be used by certain ICL interfaces
Current limitations:
Must be two single chassis, no clusters
Load balancing is active/standby per satellite, per access port planned
No user configuration sync between hosts
Configuration Differences:
The most notable changes when coming from a simple hub-and-spoke design is ICCP, and adding the satellite serial number.
Example
Router 1
redundancy
iccp
group 1
member
neighbor 172.18.0.2
!
nv satellite
system-mac <mac> (optional)
!
!
!
!
nv
satellite 100
type asr9000v
ipv4 address 10.0.0.100
redundancy
host-priority <priority> {optional)
!
serial-number <satellite serial>
!
vrf nv_mgmt
!
interface loopback 10
vrf nv_mgmt
ipv4 address 10.0.0.1
!
interface Loopback1000
ipv4 address 172.18.0.1 255.255.255.255
!
interface GigabitEthernet0/1/0/4
ipv4 address 192.168.0.1 255.255.255.0
!
interface ten 0/0/0/0
ipv4 point-to-point
ipv4 unnumbered loopback 10
nv
satellite-fabric-link [network satellite <> | satellite <>]
redundancy
iccp-group 1
remote-ports gig 0/0/0-43
!
!
!
mpls ldp
router-id 172.18.0.1
discovery targeted-hello accept
neighbor 172.18.0.2
!
!
!
router static
address-family ipv4 unicast
172.18.0.2/32 192.168.0.2
!
!
Starting in 5.1.1 we have the ability to support more than just simple hub-and-spoke. The ring topology allows for satellite chaining, cascading, and in general a more advanced satellite network.
Requirements and Limitations:
Configuration:
This is essentially the same as the dual hosts setup, but the network option must be used when entering 'satellite-fabirc-link'
This is treated as special ring and works the same way as simple ring.
The biggest difference is that in 5.1.1 cascading supports single host while simple ring does not.
Starting in 5.1.1 we have the ability to extend the ICL across an EVC. Normally an IRL is a L1 connection. This increases the flexibility of satellite by allowing for greater distances between the ASR9K host and the satellite device.
Requirements and limitations:
Configuration:
On Active-Host:
interface TenGigE0/1/0/23.200
encapsulation dot1q 200
!
nv
satellite-fabric-link satellite 200
redundancy
iccp-group 1
!
remote-ports GigabitEthernet 0/0/0-43
!
On Standby-Host:
interface TenGigE0/1/1/0.200
encapsulation dot1q 220
!
nv
satellite-fabric-link satellite 200
redundancy
iccp-group 1
!
remote-ports GigabitEthernet 0/0/0-43
!
Note: L2 cloud configuration not shown
'show nv satellite status'
Checking Version: The version of the software running on the satellite is being checked for compatibility with the running version of IOS-XR on the host
Configured Serial Number: (If configured) the serial number configured for the satellite, checked against that presented by the satellite during control protocol authentication
Configured Satellite Links: One entry for each of the configured satellite-fabric-links, headed by the interface name. The following information is present for each configured link:
Discovered Satellite Fabric Links: This section is only present for redundant satellite-fabric-links. This lists the interfaces that are members of the configured link, and the per-link discovery state.
Conflict: If the configured link is not conflicted, the satellite discovered over the link is presenting data that contradicts that found over a different satellite-fabric-link.
'show nv satellite protocol discovery'
Host IPv4 Address: The IPv4 address used for the host to communicate to this satellite. Should match the IPv4 address on all the satellite-fabric-links
For Bundle-Ether satellite-fabric-links, there are then 0 or more 'Discovered links' entries; for physical satellite-fabric-links, the same fields are present but just inline.
'show nv satellite protocol control'
Authenticating: The TCP session has been established, and the control protocol is checking the authentication information provided by the Satellite
Connected: The SDACP control protocol session to the satellite has been successfully brought up, and the feature channels can now be opened.
For each channel, the following fields are present:
Open(In Resync - Awaiting Client Resync End) The Feature Channel Owner (FCO) on the host has not finished sending data to the FCO on the Satellite. If this is the state then typically the triage should continue on the host by the owner. The owner of the Feature Channel should be contacted.
Open(In Resync - Awaiting Satellite Resync End) The FCO on the Host is awaiting information from the FCO on the Satellite. If this is the state then typically the triage should continue on the satellite.
Notes:
icpe_gco[1148]: %PKT_INFRA-ICPE_GCO-6-TRANSFER_DONE : Image transfer completed on Satellite 101
A few issues can cause this:
Conflict Messages
Examples:
BNG access over satellite is only qualified over bundle access and isn’t supported over bundle ICLs.
BNG access over ASR9k host and NCS5k satellite specifically is in the process of official qualification in 6.1.x. Please check with PM for exact qualified release.
Access bundles across satellites in an nV dual head solution are generally not recommended. The emphasis is not to bundle services across satellites in a dual head system as if they align to different hosts, the solution breaks without an explicit redundant path. An MCLAG over satellite access is a better solution there.
Bundle access over bundle fabric / ICL require 5.3.2 and above on ASR9k. For NCS5k satellite, bundle ICL including bundle over bundle is supported from 6.0.1 and nV dual head topologies are planned to be supported only from 6.1.1
MC-LAG over satellite access might be more convergence friendly and feature rich than nV dual head for BNG access from previous case studies. For non BNG access, nV dual head and MC-LAG are both possible options with any combinations of physical or bundle access and fabric.
In an MC-LAG with satellite access, the topology is just a regular MC-LAG system with the hosts syncing over ICCP but with satellite access as well. Note that the individual satellites aren’t dual homed/hosted here so there is no dual-host feature to sync over ICCP beyond just MC-LAG from CE.
As a deployment recommendation, unless ICL links (between satellite and host) are more prone to failure over access, MC-LAG might be preferable over nV dual head solution. However, if ICL links have higher failure probability and the links going down can affect BW in bundle ICL cases, then MC-LAG may not switch over unless the whole link goes down or access goes down.
Xander Thuijs CCIE#6775
Principal Engineer, ASR9000
Sam Milstead
Customer Support Engineer, IOS XR
Hi, Sam:
Do you have an update for LAGonLAG through an nvSAT? Is it commited for 5.4? Is there an ETA for 5.4?
Thanks a lot!
c.
The feature is still being tested but should be available before the end of the summer.
We are doing some software version number changing this year so whether its 5.3.2, 5.4.0, or 6.x.y is still up in the air.
Sam
thanks, sam!
Hi, Xander/Sam:
Quick question: for a dual-home environment, when the active linkA goes down from sat-to-hostA, the standby linkB from sat-to-hostB takes over. What is expected to happen when linkA recovers? What we've seen is that it doesn't revert back to linkA. Is there a way to control this and force a 'revertive' behavior? What would be the command for 5.1 to enforce this in case it can be controlled?
Thanks,
c.
Xander/Sam:
on the configuration guide for nv sat, there is a comment on dual-homing that has me confused a little, so i thought i'd ask you guys.
i have two nv sats connected to same hostA and hostB. if i use a different iccp-group for each, i see a message that looks like this:
Status: Configuring Satellite; Conflict: Global limit of 1 redundancy-group for nV
if i use the same iccp-group for both, everything looks normal. so, the limitation would be using same iccp-group for all nv sats connected to same hosts?
what if i introduce a hostC? all nv sats between A and C would have to use a separate iccp-group than A and B, but as long as they all use the same group it'll work?
thanks,
c.
Hello X,
I hope you are great !! :) We are running out time configuring a POC, and It could be great to have a little help.
So, just to confirm:
a. It is possible to have a single ring (3 ASR9000v) connected to:
- One end facing a single ASR9001
- The other end facing a cluster ASR9001
b. Regarding the ICCP Link, how do you establish it ?
c. We have a ASR9001 cluster, the OEBC links are ok, but when We connect the IRL Links, the secondary-box´s interfaces start to reload every 5 minutes. We tried with 4 different boxes, same issue. Could It be a license issue, a bad connection between IRL interfaces, or something else ? Do you have any clue ? (attached a log, maybe It is incomplete)
Cheers man, thanks
Mariano
hey mariano!
for "a"; somewhat of an unusual design as you probably want to home the statellite to the cluster as that is already redundant. while it should technically just work, I am sure that we dont test for this scenario per-se which only means that there may be a discrepancy I am not aware of, but it is worth the try I would say
in regards to ICCP; that pertains to MCLAG where would you be running MCLAG? off the statellite ports or have an MCLAG bundle with members on 2 different satellites?
the crash seems to be an envmon crash, which sounds somehwat familiar, it looks like from the dumper you're not running any smu's?
would probably be best to run the 513 latest smu's and retest.
You may be hitting: CSCur13869 for which there is a smu on CCO.
cheers!! :)
xander
Another excellent post, Xander! One question about dual home - can we add an additional 10gig pair? Basically, we'd have a single 9000v satellite with 2 each 10gig going to one 9k chassis, and the other 2 each 10gig going to the second 9k chassis. The active 'bundle' from each 9000v would carry traffic for all satellite interfaces 0-43. Diagram attached.
Or, am I limited to doing a single 10gig to each 9k? I would have to assign a subset of the 9000v interfaces to the links to the 9k and repeat the procedure twice (0-21 one one pair of 10gig going to the 9k pair and 22-43 on the other pair of 10gig going to the 9k pair)?
hi ben,
thanks!! you can have a bundle on either the ICL or on the access, not on both.
so if you dont have bundles on the satellite interfaces itself, you should be able to make a bundle on the ICL.
regards!
xander
thanks for the update, xander.
one other question - i believe there is a limit of one dual homed 9000v per 9k pair. is that correct? if i connect one 9000v to a pair of 9ks, i cannot do another in the same fashion?
thanks again.
-ben
For Management configuration what is the logical next step to add multiple 9000nv Satellites to a single ASR9k if you do not use Auto-IP. Example see below configuration.
To add a second independent 9000nv lets say it is 600 would we just add the next sequential Availble IP Address in the Existing Private vrf NV_MGMT_VRF ? as shown below, rather than creating a second Private VRF ?
router(config)# nv satellite 600
router(config)# ipv4 address 10.0.0.3 / 24
Satellite Management using private VRF
You can use a special private VRF instead of the global default routing table, to configure the loopback
interface and ICLs used for satellite management traffic. IP addresses in this VRF will not conflict with
any other addresses used on the router.
router(config)# vrf NV_MGMT_VRF
router(config)# address ipv4 unicast
router(config)# interface Loopback 1000
router(config)# vrf NV_MGMT_VRF
router(config)# ipv4 address 10.0.0.1 / 24
router(config)# interface TenGige 0/1/0/3
router(config)# vrf NV_MGMT_VRF
router(config)# ipv4 point-to-point
router(config)# ipv4 unnumbered Loopback 1000
router(config)# nv
router(config-nv)# satellite-fabric-link satellite 500
router(config-nv)# remote-ports GigabitEthernet 0/0/28-39
router(config)# nv satellite 500
router(config)# ipv4 address 10.0.0.2 / 24
You have to use the same VRF for all satellites connected off a single ICL.
Please note that we (Cisco) strongly recommend the use of Auto-IP feature. Is there a particular reason why you prefer static configuration?
/Aleksandar
We did not use auto IP because we are in 4.2.3 which I believe did not support auto IP , we are upgrading to 5.1.3 but the 9000nv is being installed prior to the upgrade
Hi,
as for the licensing: I was surprised that when using simple ring, a host license for each satellite is needed on each host. E.g. a simple ring with three satellites requires six A9K-NVSAT1-LIC licenses.
Maybe this piece of information should be added to the "Licensing" section.
Thank you!
Thorsten
Hey Xander,
Great article as always. Have a question regarding MTU over Satellite ICL. How many bytes is the NV-Tag that is added to ethernet frames. If we send an ethernet frame of 9216 into the satellite port it gets dropped. It appears the largest frames we can pass through are 9198. Is there a way to increase the MTU of the ICL to work around this?
thanks,
Jon
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: