Symptoms  When VSM services part fails, the failure will not propagate to ABF, which will result in black holing of the traffic instead of switching the traffic to backup VSM, if one exists Solution To track the reach-ability between Ingress service App interface and Egress traffic port using IPSLA and trigger the ABF when this reach-ability breaks.   Details   Consider a router with VSM and CGN services running on both of them. We have applied an ABF on Ingress port with first next hop pointing to Ingress service App in VSM1 and second next hop pointing to Ingress service App in VSM 2. At any instance, when next hop 1 becomes unreachable, the ABF will start diverting the traffic to service App in VSM2. Hence black holing of traffic will be avoided.    Each VSM has service and Infra part. There are situations where a service part of the VSM will not get propagated to infra part of VSM, which results in ABF been not notified of the next hop failure. So ABF will not switch the traffic to next hop, instead will continue to forward traffic to VSM1 which will result in traffic dropped as service part is no longer in running state to process the traffic.    To mitigate this scenario, one of the approach it to implement IPSLA which will track the path from Ingress service App to Egress traffic port. IPSLA used ICMP to track the path. If any failure happens in service part, this path will break and IP reachability goes down. Hence when service part of VSM fails, the IPSLA catches the failure. The failure will be propagated to ABF from then on and action will be taken accordingly which will be switchover of traffic from nexthop1 to next hop 2. Even after the failure, the IPSLA continues to track the path and once it comes back again, ABF will switch traffic back to nexthop1. Thus the traffic black holing due to services part failure is mitigated.    Following figures give a clear explanation on packet flow when VSM1 is active and when VSM1 fails. Also it gives explanation where the IPSLA is triggered. After these figures, the configuration examples are provided, which will help in implementing the solution without any hick ups.   This solution is verified on ASR9K routers with VSM running CGN services. We need to define one IPSLA track per VSM card using any one service App. The same can be used for tracking all ABFs associated with that card   Note: Please refer to following cisco support forum page for detailed explanation on CGN on VSM with HA scenarios. https://community.cisco.com/t5/xr-os-and-platforms/ism-100-to-vsm-500-migration-and-redundancy-managment-guide-with/m-p/2876457    IPSLA Tracking Enabled between service Ingress Port and Egress Traffic Port: IPSLA is run between Ingress service App port of VSM1 and egress traffic port. IPSLA used ICMP for tracking On failure ABF will be notified, which results in traffic switchover according to the ABF rule defined   Traffic Path When VSM 1 is Active: Traffic enters Ingress traffic port. Then based on ABF applied on Ingress traffic port, which has next hop IP as Ingress Service App IP, also based on IPSLA which doesn’t report any failure, traffic enters service App 1 in VSM1. Then traffic passes to Egress service App on VSM1 along with NAT been applied. Then traffic exits out of Egress traffic port based on Routing information available. Example configuration: Ingress traffic port configuration: RP/0/RSP0/CPU0:Hulk#sh running-config interface bundle-ether 400.101 interface Bundle-Ether400.101  ipv4 address 10.22.231.2 255.255.255.252  encapsulation dot1q 101  ipv4 access-group VNAT1_ABF ingress ! Egress traffic port configuration: interface TenGigE0/3/0/4  ipv4 address 22.1.1.1 255.255.255.0 ! Ingress service App configuration: RP/0/RSP0/CPU0:Hulk#sh running-config interface serviceApp 1  interface ServiceApp1  vrf SVSM-1  ipv4 address 10.22.241.14 255.255.255.252  load-interval 30  service cgn VSM-1 service-type nat44 ! VRF configuration for ingress service App:   RP/0/RSP0/CPU0:Hulk#sh running-config vrf SVSM-1 vrf SVSM-1  description <<inside vrf1 on VSM-1>>  address-family ipv4 unicast   import route-target    20:1   !   export route-target    20:1   !  ! ! Static route to Traffic Egress port under Ingress next Hop: RP/0/RSP0/CPU0:Hulk#sh running-config router static vrf SVSM-1 router static  vrf SVSM-1   address-family ipv4 unicast    10.1.1.0/24 vrf default 10.22.231.1    22.1.1.1/32 ServiceApp1   !  ! ! IPSLA Configuration: RP/0/RSP0/CPU0:Hulk#sh running-config ipsla ipsla  operation 1000   type icmp echo    vrf SVSM-1    source address 10.22.241.14    destination address 22.1.1.1    timeout 2000    frequency 5   !  ! reaction operation 1000   react timeout    action logging   !  ! schedule operation 1000   start-time now   life forever  !      Track configuration: RP/0/RSP0/CPU0:Hulk#sh running-config track track TSA1  type rtr 1000 reachability  delay up 5  delay down 5 ! ABF configuration (ABF is attached to ingress traffic port): RP/0/RSP0/CPU0:Hulk#sh running-config ipv4 access-list VNAT1_ABF ipv4 access-list VNAT1_ABF  10 permit ipv4 10.1.1.0/24 any nexthop1 track TSA1 vrf SVSM-1 ipv4 10.22.241.13 nexthop2 vrf SVSM-33 ipv4 10.22.241.141  20 permit ipv4 10.1.2.0/24 any nexthop1 track TSA1 vrf SVSM-3 ipv4 10.22.241.21 nexthop2 vrf SVSM-35 ipv4 10.22.241.149  30 permit ipv4 10.1.3.0/24 any nexthop1 track TSA1 vrf SVSM-5 ipv4 10.22.241.29 nexthop2 vrf SVSM-37 ipv4 10.22.241.157  40 permit ipv4 10.1.4.0/24 any nexthop1 track TSA1 vrf SVSM-7 ipv4 10.22.241.37 nexthop2 vrf SVSM-39 ipv4 10.22.241.165  50 permit ipv4 10.1.5.0/24 any nexthop1 track TSA1 vrf SVSM-9 ipv4 10.22.241.45 nexthop2 vrf SVSM-41 ipv4 10.22.241.173  60 permit ipv4 10.1.6.0/24 any nexthop1 track TSA1 vrf SVSM-11 ipv4 10.22.241.53 nexthop2 vrf SVSM-43 ipv4 10.22.241.181  70 permit ipv4 10.1.7.0/24 any nexthop1 track TSA1 vrf SVSM-13 ipv4 10.22.241.61 nexthop2 vrf SVSM-45 ipv4 10.22.241.189  80 permit ipv4 10.1.8.0/24 any nexthop1 track TSA1 vrf SVSM-15 ipv4 10.22.241.69 nexthop2 vrf SVSM-47 ipv4 10.22.241.197  1000 permit ipv4 any any !                              IPSLA statistics: RP/0/RSP0/CPU0:Hulk#sh ipsla statistics Entry number: 1000     Modification time: 16:15:15.275 IST Fri May 20 2016     Start time       : 16:15:15.276 IST Fri May 20 2016     Number of operations attempted: 18435     Number of operations skipped  : 0     Current seconds left in Life  : Forever     Operational state of entry    : Active     Operational frequency(seconds): 5     Connection loss occurred      : FALSE     Timeout occurred              : FALSE     Latest RTT (milliseconds)     : 47     Latest operation start time   : 17:51:25.481 IST Sat May 21 2016     Next operation start time     : 17:51:30.481 IST Sat May 21 2016     Latest operation return code  : OK     RTT Values:       RTTAvg  : 47         RTTMin: 47         RTTMax : 47              NumOfRTT: 1          RTTSum: 47         RTTSum2: 2209   Results : VSM HA will induce 15sec of delay while convergence if CGN APP fails. IPSLA solution can help us reduce the SWO time considering IPSLA probes tracking.   Note: Probe timers should not be less than 10 to avoid building stress conditions.     ... View more

Condition: In certain network scenarios where network admin require correctly process exception packets in MAPT inline solution than below document help community to understand ASR9k In-line CGN solution using VSM card. Solution:  IN-LINE MAP-T / E deployment using 3 rd generation ASR9k line cards and using ASR9k as border relay router.   Details:   MAP-T Introduction Enables the delivery of IPv4 services over an IPv6 infrastructure through the use of stateless translation (MAP-T) Uses Address and Port (A+P) architecture to enable the sharing of IPv4 address by distributing the port-space, i.e. Embeds IPv4 address + Port into IPv6 address. Use-Cases: Residential wire line operators constrained by run-out with desire to deploy an IPv6-only access/aggregation Limitation: Provider make sure CPEs are capable of doing NAT44 operations. Understanding MAP- T Exception Handling The MAP-T Exception Handling with VSM feature handles fragmented packets, TCP, MSS clamping, path MTU, UDP and ICMP packets. Depending on the header details of the fragmented packets, the CGN application directly interacts with the line cards to process or forward the packets to VSM.   Note: When MAP-T is configured for exception handling with VSM, the static route details are automatically configured with Service App details. No manual configuration is required to configure the static routes.   Reference manuals & Hardware Supported Internet Drafts : draft-ietf-softwire-map - RFC 7599 draft-ietf-softwire-map-dhcp - RFC 7598 draft-ietf-softwire-stateless-4v6-motivation Chassis : ASR90xx and 99xx chassis RP : RSP-880-SE/TR Line card : 8*100G (Tomahawk) or MoD400 (for handling inline traffic) Services : A9K-VSM-500 (VSM Card) (For Handling Fragmented, IPv6 Extension headers Containing Packets, TCP Max Segment Size, MTU checks, ICMP v4/v6) MAP-T Topology Overview   MAP Components MAP CPE : Performs NAT44 (translates source IPv4 address and TCP/UDP port to assigned range) Performs stateless translation (MAP-T) of IPv4 to IPv6 MAP Border Relay : Managed by SP deployed at edge with IPv4 and IPv6 Internet connectivity Performs translation based on predefined rules Port-set : Separate transport layer port space Port-set ID : Allows the identification of set of ports assigned to a CE Note: Provider make sure that customer CPE is capable to perform NAT44 operations. Without such capable CPEs, this solution is not possible.   Types of exception packets handled   In 6.2.1 release, we support functionality of Map-T translations on Tomahawk line card and exception packets by VSM. ICMP : In Inline MAPT, without VSM echo request and reply was handled by Tomahawk. Now all ICMP traffic is handled by VSM. ICMP error message All Destination unreachable (except code 14 in v4tov6) Time exceeded All Parameter Problem( except code 1 in v4tov6 and code 2 in v6tov4) ICMP query messages  V4/V6 fragmented packets. TCP Maximum Segment Size : TCP packets with SYN flag set is handled in VSM for recalculation of MSS. Path MTU Checks : IPv4 to IPv6 path MTU – packets with DF bit set and packet length > MTU IPv6 to IPv4 path MTU – packet length > MTU IPv4 options : Packets with LSR and SSR IPv4 options will be dropped and ICMPv4 error generated IPv6 extended headers Hop-by-Hop Destination Header Routing header Fragment header Authentication header Note: Only default vrf is supported for MapT exception handling feature Configuration Interface ServiceInfra : Configure serviceInfra interface Interface ServiceApp : Configure service App interfaces for each ipv4 and ipv6 Service cgv6 <service-name> :  Configure CGv6 service name service-location preferred-active node-id : mention VSM location for handling exception traffic service-inline interface-name/s : mention tomahawk ingress and egress interfaces service-type map-t-cisco <instance-name> : Configure map-t instance Within each map-T instance, parameters to be configured are mentioned in following slide Under each Map-T instance, configure : address-family : ipv4 and corresponding serviceApp interface ipv6 and corresponding serviceApp interace Tcp mss and path-mtu value (non-mandatory field, default MTU is 1500) cpe-domains for : ipv4 prefix length value ipv6 vrf default ipv6 prefix length value Sharing ratio and contiguous-ports number Single or multiple cpe-domain/s : cpe-domain-name <domain-name1> ipv4-prefix <ipv4 address/prefix> ipv6-prefix <ipv6 address/prefix>…… Single ext-domain : Ext-domain-name <domain-name> ipv6-prefix <ipv6 address/prefix> ipv4-vrf default Sample Config Show run interface serviceapp *: interface ServiceApp1  ipv4 address 40.40.40.1 255.255.255.0  service cgv6 cgn123 service-type map-t-cisco ! interface ServiceApp2  ipv6 address 2345::8/64  service cgv6 cgn123 service-type map-t-cisco ! Show run interface serviceinfra * interface ServiceInfra1  ipv4 address 1.1.1.1 255.255.255.252  service-location 0/3/CPU0 ! Show run interface TenGigE0/2/0/5/9: interface TenGigE0/2/0/5/9  ipv4 address 4.4.4.1 255.255.255.0  ipv6 address 2001::1/64  load-interval 30 ! Service CGv6 Configuration: service cgv6 cgn123  service-location preferred-active 0/3/CPU0  service-inline interface TenGigE0/2/0/5/9  service-type map-t-cisco map1   address-family ipv4    interface ServiceApp1    tcp mss 335    path-mtu 1200   !   address-family ipv6    interface ServiceApp2    tcp mss 1254    path-mtu 1500   !   cpe-domain ipv4 prefix length 24   cpe-domain ipv6 vrf default   cpe-domain ipv6 prefix length 48   sharing-ratio 256   contiguous-ports 16   cpe-domain-name cpe0 ipv4-prefix 192.1.1.0 ipv6-prefix 2301:0:1122::   ext-domain-name ext1 ipv6-prefix 6301:d01:1122::/48 ipv4-vrf default  ! ! Show Commands sh int te 0/2/0/5/9 accounting : TenGigE0/2/0/5/9   Protocol                             Pkts In                 Chars In                              Pkts Out              Chars Out   IPV4_UNICAST               119429796         141175259448                 57549767      25454289540   IPV6_UNICAST               57573860           26617932084                   119419021     144497009720   IPV6_MULTICAST       7              472                                      0                           0   IPV6_ND                          20                         1600                                   24                         1920 sh int serviceapp * accounting ServiceApp1   Protocol                           Pkts In         Chars In     Pkts Out        Chars Out   IPV4_UNICAST             8141          3485668    193852302     229133420964 ServiceApp2   Protocol                             Pkts In         Chars In                       Pkts Out        Chars Out   IPV6_UNICAST        193856659     234566557390         8148                        3718740   show cgv6 map-t-cisco map1 statistics Map-t-cisco IPv6 to IPv4 counters: ====================================== Translated Udp Count: 652567655 Translated Tcp Count: 0 Translated Icmp Count: 0 PSID Drop Udp Count: 0 PSID Drop Tcp Count: 0 PSID Drop Icmp Count: 0 Map-t-cisco IPv4 to IPv6 counters: ====================================== Translated Udp Count: 0 Translated Tcp Count: 0 Translated Icmp Count: 0 PSID Drop Udp Count: 0 PSID Drop Tcp Count: 0 PSID Drop Icmp Count: 0 Map-t-cisco exception IPv6 to IPv4 counters: ====================================== TCP Incoming Count: 0 TCP NonTranslatable Drop Count: 0 TCP Invalid NextHdr Drop Count: 0 TCP NoDb Drop Count: 0 TCP Translated Count: 0 TCP Psid Drop Count: 0 UDP Incoming Count: 126796 UDP NonTranslatable Drop Count: 0 UDP Invalid Next Hdr Drop Count: 0 UDP No Db Drop Count: 0 UDP Translated Count: 126796 UDP Psid Drop Count: 0 ICMP Total Incoming Count: 0 ICMP No DB Drop Count: 0 ICMP Fragment drop count: 0 ICMP Invalid NxtHdr Drop Count: 0 ICMP Nontanslatable Drop Count: 0 ICMP Nontanslatable Fwd Count: 0 ICMP UnsupportedType Drop Count: 0 ICMP Err Translated Count: 0 ICMP Query Translated Count: 0 ICMP Psid Drop Count: 0 Map-t-cisco IPv6 to IPv4 counters: ====================================== Translated Udp Count: 652567655 Translated Tcp Count: 0 Translated Icmp Count: 0 PSID Drop Udp Count: 0 PSID Drop Tcp Count: 0 PSID Drop Icmp Count: 0 Map-t-cisco IPv4 to IPv6 counters: ====================================== Translated Udp Count: 0 Translated Tcp Count: 0 Translated Icmp Count: 0 PSID Drop Udp Count: 0 PSID Drop Tcp Count: 0 PSID Drop Icmp Count: 0 Map-t-cisco exception packets IPv4 to IPv6 counters: ====================================== TCP Incoming Count: 0 TCP No Db Drop Count: 0 TCP Translated Count: 0 TCP Psid Drop Count: 0 UDP Incoming Count: 2134370 UDP No Db Drop Count: 0 UDP Translated Count: 2134370 UDP FragmentCrc Zero Drop Count: 0 UDP CrcZeroRecy Sent Count: 0 UDP CrcZeroRecy Drop Count: 0 UDP Psid Drop Count: 0 ICMP Total Incoming Count: 0 ICMP No Db Drop Count: 0 ICMP Fragment drop count: 0 ICMP UnsupportedType Drop Count: 0 ICMP Err Translated Count: 0 ICMP Query Translated Count: 0 ICMP Psid Drop Count: 0 Subsequent Fragment Incoming Count: 264661768 Subsequent Fragment No Db Drop  Count: 0 Subsequent Fragment Translated Count: 264661768 Subsequent Fragment Drop Count: 0 Subsequent Fragment Throttled Count: 0 Subsequent Fragment Timeout Drop Count: 36 Subsequent Fragment TCP Input Count: 0 Subsequent Fragment UDP Input Count: 2134370 Subsequent Fragment ICMP Input Count: 0 Options Incoming Count: 0 Options Drop Count: 0 Options Forward Count: 0 Options No DB drop Count: 0 Unsupported Protocol Count: 0 ICMP generated counters : ======================= IPv4 ICMP Messages generated count: 0 IPv6 ICMP Messages generated count: 0 Troubleshooting tips Checking core file at VSM: RP/0/RSP0/CPU0:ROUTER#virtual-service connect name Mapt console node 0/1/CPU0 Trying 192.0.131.3... Connected to 192.0.131.3. Escape sequence is '^^e'. Red Hat Enterprise Linux Server release 5.3 (Tikanga) Kernel 2.6.18-128.el5 on an x86_64 localdomain login: root Password:rootroot [root@localhost ~]# cd VSM [root@localhost VSM]# pwd /root/VSM [root@localhost VSM]# ls -l total 2445520 <> -rw------- 1 root root 425799680 Oct 5 23:16 core.7088 Checking VQI/UIDB Configuration in NP Datastructures show uidb data location 0/4/CPU0 serviceApp 1 extension show cef adjacency serviceApp 1 hardware ingress detail location 0/0/CPU0 | inc sfp sh controllers pm vqi location 0/2/CPU0 Checking Routes/PBR entries for CPE IPv4 and EXT IPv6 show policy-map transient type pbr  show route show route ipv6 Checking for VQI Related Drops sh controllers fabric fia drops ingress location 0/0/CPU0 | inc Vqi Checking Ucode Datastructures related to CGN show controllers np struct 113 detail all-entries np0 location 0/0/CPU0  show controllers np struct 114 detail all-entries np0 location 0/0/CPU0  Checking P2MP Node counter in VSM and in XR To connect to virtual host use root/rootroot . RP/0/RSP0/CPU0:ROUTER#show virtual-service list Virtual Service List: Service Name Status Package Name Node Name ______________________________________________________________________________ Mapt Activated asr9k-vsm-cgv6-6.2.1.00- 0/1/CPU0 RP/0/RSP0/CPU0:ROUTER#virtual-service connect name Mapt console node 0/1/CPU0 Trying 192.0.131.3... Connected to 192.0.131.3. Escape sequence is '^^e'. Red Hat Enterprise Linux Server release 5.3 (Tikanga) Kernel 2.6.18-128.el5 on an x86_64 localhost.localdomain login: root Password: rootroot [root@localhost ~]# cd /var/log/cgv6/ [root@localhost cgv6]# pwd /var/log/cgv6 [root@localhost cgv6]# p2mp_debugger Dump options: 0 -> Policy 1 -> Main DB 2 -> User DB 3 -> Hashes DB 4 -> VRF Map 5 -> Summary DB 6 -> Dump Statistics 7 -> Clear Statistics 8 -> Dump node counter 9 -> Clear node counter 10 -> Dump CNAT counter 11 -> Dump Virtual Assembly (VA) 12 -> Show Configuration 13 -> Show Netflow V9 Configuration 14 -> Show Inside VRF Information 15 -> Show Outside VRF Information 16 -> Logging util test 17 -> Configure PD/PI/L2 Debug Level 18 -> Dump PD/P2/L2 Debug Level 19 -> Set Traffic Flags 22 -> Dump Main DB Summary 23 -> Dump User DB Summary 25 -> Dump Timeout DB Summary 27 -> Set bulk size for nat44 28 -> Dump bulk port statistics 29 -> Clear bulk port statistics 30 -> Show bulk port allocation for subscriber Enter dump option <0-30>:8 <------- option 8 to dump cgn node counters Enter the coremask in hex: ffffffffffff Dump_option: 8, core_mask : FFFFFFFFFFFF [root@localhost cgv6]# Core1: NODE Counter dump ----------------------------------------- Node Counters ----------------------------------------- vsm/inject injected and forwarded: 5965 ------------------------------- vsm_decode Forwarded to INFRA: 3501 Forwarded to MAPT_CISCO: 2464 ------------------------------- vsm_infra_classifier Infra to CLI: 8250 646 Infra to Data path Test: 2855 ------------------------------- vsm_ha HA Packets(DP TX) injected: 2855 ------------------------------- vsm_infra_l3_tx Infra L3 Tx injected and forwa: 646 Forwarded for FIB lookup: 2855 ------------------------------- vsm_xlat_classifier xclsfr6 v4 tcp frag: 2464 ------------------------------- cnat_cli_input CNAT config messages processed: 646 ------------------------------- cnat_db_scanner Scan timer callback invoked: 1423132 ------------------------------- xlat_v4_to_v6_tcp Xmit - v4 to_v6 tcp: 2464 ------------------------------- xlat_v4_frag v4_frag_tcp: 2464   Scale Maximum of 32 domain definitions/MapT instances per VSM. Maximum of 256 CPE domains per MapT instance. Maximum of 1024 CPE domains across all MAP-T instances per VSM Single EXT domain per MapT instance   Exception Traffic Flow:   Note:   Detail Config sample information is available on below CCO link:   https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r6-2/cg-nat/configuration/guide/b-cgnat-cg-asr9000-62x/b-cgnat-cg-asr9000-62x_chapter_0111.html ... View more