cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

ASR9000/XR: Local Packet Transport Services (LPTS) CoPP

41745
Views
16
Helpful
72
Comments

Introduction

IOS devices have the concept of control plane policing. IOS-XR doesn't use that concept but instead uses a very comprehensive and powerful Local Packet Transport Services. In this document it is explained how LPTS works and how you can work with it, monitor and verify it.

Core Issue

LPTS is the concept of reflexive ACL's, punt policers and has an "internal" FIB or iFIB that directs certain packets to various nodes. IOS-XR can handle certain traffic on the linecard (such as BFD, Netflow and ARP) and these packets are instructed by LPTS to be handled by the local CPU rather then the RSP CPU.

At the same time, there are ACL's in place that allow for instance the punting of Telnet traffic and then per host if configured so, but another component of LPTS called MPP, the Management Plane Protection.

Generally, the default values for LPTS provide the level of protection you are after. However there are some rare circumstances whereby you want to tune the values of LPTS in order to get the service levels you need. LPTS is very dynamic in nature and pierces holes into the protection system as particular items are configured.

The LPTS policers work on a per NP basis. So if the LPTS police value is set to 1000pps that means that every NP on the LC can punt with 1000pps to the RSP CPU or LC CPU. This is something to take into consideration when evaluating the A9K-8T-x cards who have 8 NPU's per LC.

Take extreme care when changing the LPTS policer values.

High level overview

From a birds eye view, LPTS looks like this:

overview.jpg

The NPU has a table that tells it where to send packet to (LC or RSP) as part of the "internal FIB" or iFIB. These packets are punted at a pre-defined rate, they can be tuned in XR release 4.x and later. Also in the TCAM which is used in the ASR9K for ACLs (amongst others), are lists that define which packets we want to allow and not. This will be discussed in the MPP section of this document.

LPTS is composed of a (set of) dynamic ACL's (which are created as part of user configuration dynamically, or automitcally inserted as peerings establish), an internal "routing table" (iFIB) and a set of policers for different punt reasons.

LPTS Characteristics

0) LPTS has Hardware policers on line cards to limit traffic sent to local or remote nodes
0) LPTS entries in TCAM classifies packets to select a policer to apply
0) The policer value can be tuned to 0 (to drop all packet matching classification criteria)
0) Polices on protocol (BGP, OSPF, SSH) and flow state (BGP established, BGP configured, and BGP listen)
0) Policing done on the LC Hardware ASIC before packets hit RP/LC CPU
0) Filters are automatically and dynamically installed by the IOS XR infrastructure

So for-me packets are undergoing the Pre iFIB classification and policing upon which they are directed by the iFIB, which is the second level of filtering to the destination node.

LPTS Firewalling

One of the great strenghts with LPTS is the dynamic ACL creation. This is configuration driven and no user intervention is required.

In addition to that, LPTS has different flow categories based on the state of the protocol. For instance, BGP has 3 different states:

  1. Unknown
  2. Configured
  3. Established

Unknown is the flow whereby we have TCP port 179 traffic, but we have no neighbor configured from that source. Policed very heavily.

Configured is the entry whereby we know the source address of the peer, but the session is not yet established (no known source port from the peer), Policed moderately.

Established is where we have all the L3 and L4 data from the session. Lightly policed.

The entries for configured is driven by the configuration of the neighbor statement under the router BGP section.

Established is dynamically inserted when the peer establishes.

You could theoretically police the unknown to a rate of zero.

Example:

Router bgp

   neighbor 192.168.1.1

   …

!

The following table can be seen with the output of the command:

show lpts pifib hardware entry brief loc 0/3/cpu0 | i 179

LocalPortRemotePortRateState
any179ANYANY100unknown
any179192.168.1.1ANY1,000configured
192.168.1.2179192.168.1.1222310,000established

If you use the command

RP/0/RSP0/CPU0:A9K-TOP#show lpts pifib hardware entry location 0/3/CPU0 | be 33.33.1  

You can check the detailed entry of the PiFIB (policer)

Source IP         : 33.33.1.1                        the remote address

Is Fragment       : 0                                fragments allowed

Interface         : any                              expected source interface

M/L/T/F           : 0/IPv4_STACK/0/BGP-known

DestNode          : 48                               where the packets are sent to

DestAddr          : 48

L4 Protocol       : TCP

TCP flag byte     : any                              additional security checks at TCP level

Source port       : Port:179

Destination Port  : 11293

Accepted/Dropped  : 117866/0                         packets accepted and denied

# of TCAM entries : 1                                number of tcam entries burnt for this PiFIB entry

HPo/HAr/HBu/Cir   : 1924676/2500pps/2500ms/2500pps

State             : Entry in TCAM                    status of the entry


Configuring LPTS police rates

You can configure the LPTS Policers on a PiFIB bases and also the punt policers can be adjusted.

The following commands apply. Note that this is on a per linecard basis. All NPU's on that linecard will get reconfigured.

RP/0/RSP0/CPU0:A9K-BNG(config)#lpts punt police location 0/0/CPU0 protocol ?

  arp              ARP packets

  bfd              Bidirectional Forwarding Detection packets

  cdp              Cisco Discovery Protocol packets

  cfm              Connectivity Fault Management Protocol packets

  cgmp             Cisco Group Management Protocol packets

  dhcp             Dynamic Host Configuration Protocol packets

  efm              Ethernet in the First Mile Protocol packets

  igmp-snoop       Internet Group Management Protocol Snoop packets

  ipiw-arp         L2VPN IPIW ARP packets

  ipv4             IPv4 packets

  ipv6             IPv6 packets

  lacp             Bundle Protocol packets

  mofrr            Multicast-only FRR packets

  mpls             MPLS punt packets

  mstp             Multiple Spanning Tree Protocol packets

  mvrp             Multiple VLAN Registration Protocol packets

  ppp              Point-to-Point Protocol packets

  pppoe            Point-to-Point Protocol over Ethernet packets

  rarp             Reverse ARP packets

  vccv             Virtual Circuit Connection Verification packets

  vidmon           Video Monitoring packets

  vidmon-flow-add  Video Monitoring flow add packets

Exception packets can be reconfigured by the following command: lpts punt police location 0/0/CPU0 exception

Glean adjacency or ACL-deny packets can be tuned for instance via that command.

The PIFIB can be reconfigured via the following commands:

RP/0/RSP0/CPU0:A9K-BNG(config)#lpts pifib hardware ...

  • In there you can enter the linecard you wish to specifically reconfigure
  • The policer flow values
  • And the TCAM entries (this is new in XR420)   
    • As you've seen LPTS can dynamically create "ACL" entries for dynamic firewalling and for MPP. This command limits the number of TCAM entries that LPTS can use so that space is available for other purposes such as regular ACL's, QOS matching, EFP matching etc.

LPTS static-police and police differences

The command “police” is used to check policer values, accept/drop counts for packets matching LPTS TCAM(mostly L3 packets) entries whereas “static-police” is used to check policer values.

Accept/drop counts for packets matching static punt reasons programmed in search structures (Mostly L2 and exception packets).

policer” is for dynamic flows (like BGP, OSPF, etc protocols directed  by RSP)

static-policer” is for pseudo Static flows (like BFD, CFM directed by the LC) These are hard-coded and include Exception processing packets.

There is a CLI to change few of the exception processing as well (for e.g. ICMP unreachable)

Monitoring LPTS

LPTS is not SNMP enabled (request has been filed and is in the works, no target release defined at time of writing). Though there are very inventive ways to monitor LPTS and generate alerts. There is a TCL script that you can use with EEM in order to get some level of alerting.

Attached to this article is the script package and here is how you set it up:

event manager environment EEM_LPTS_CHECK_INTERVAL 300

event manager environment EEM_LPTS_CHECK_FLOWTYPES BGP-known *

event manager environment EEM_LPTS_CHECK_LOCATIONS 0/0/CPU0 0/4/CPU0

event manager environment EEM_LPTS_CHECK_THRESHOLD 1 50%

event manager directory user policy disk0:/scripts/

event manager policy lpts-threshold-alerting.tcl username scripts

How to clear LPTS statistics

LPTS stats cannot be cleared by LPTS commands or qos counter clearing.

You can clear LPTS stats by clearing hte np controller stats:

"clear controllers np counters all location <>”

MPP: Managed Plane Protection

In the standard configuration all interfaces have access to the Telnet, SSH and SNMP daemons.

Inband vs Out of band

All linecard interfaces are designated to be inband, meaning they can transport user traffic as well as management traffic.

The mgmt interfaces on the RSP are designated out of band. This means that they can't transport user traffic but only management traffic.

Out-of-band interfaces can't "speak" to other interfaces as they are desginated for managment traffic. So eventhough there is a route in the system that would send traffic out of the mgmt interface, Fabric enabled interfaces on the LC can't

Here an example of out of band and the restrictions that it imposes on the forwarding

oob.jpg

Configuring MPP

By default when the service is configured, there are no mpp restrictions. All interfaces are able to accept the mgmt traffic for the service you defined. For example, when the telnet server is configured, LPTS reports the following binding:

RP/0/RSP0/CPU0:A9K-BNG#show lpts bindings brief | i (any.23 )

Tue Feb 28 12:00:55.195 EDT

0/RSP0/CPU0 TCP  LR IPV4 TCP    default   any          any,23 any

This means that every for me packet with port 23 as the dest port will get serviced.

Now when configuring MPP the bindings output changes:

control-plane

management-plane

  inband

   interface TenGigE0/1/0/0

    allow Telnet peer

     address ipv4 3.3.3.3

     address ipv4 5.5.5.0/28

    !

   !

   interface GigabitEthernet0/0/0/10

    allow Telnet

   !

  !

!

In this configuration example I am designating two interfaces as inband, so they will still be able to forward transient traffic and allow inbound telnet traffic. At the same time I allow telnet from any host on Gig0/0/0/10 and only telnet from a few peers on Te0/1/0/0.

The LPTS bindings are dynamically changed as per following output:

RP/0/RSP0/CPU0:A9K-BNG#show lpts bindings brief | i (any.23 )

Tue Feb 28 12:06:48.339 EDT

0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Gi0/0/0/10   any,23 any                 << Any source can access my telnet on this intf

0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Mg0/RSP0/CPU0/0 any,23 any            << Dedicated inband

0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Te0/1/0/0    any,23 3.3.3.3          << /32 host access for telnet on dedicated intf

0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Te0/1/0/0    any,23 5.5.5.0/28    << Hosts from this subnet on this intf

Powerful eh!?!

We can also look at the pre internal fib (piFIB) and check the entries there:

RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware entry bri location 0/1/cpu0 | i (.23 )

Tue Feb 28 12:27:46.389 EDT

7      IPV4 default      TCP    Te0/1/0/0       LU(48) any,23 3.3.3.3,any

10     IPV4 default      TCP    Te0/1/0/0       LU(48) any,23 5.5.5.0/28,any


Decoding the Destnode in LPTS entries

In the example above you see the following detail: LU(48). This section explains that number and detail.

The LU means local unicast fabric. The 48 is a very interesting number.

The device that this output is taken from is an ASR9010. Which has 8 LC slots and 2 RSP slots. On both sides of the RSP's in the middle

are the 4 LC's

If I were to decode the 30 into binary it looks like this:

+---+---+---+---+---+---+---+---+

| 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |     Position

+---+---+---+---+---+---+---+---+

| 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |     Bit value for Decimal 48

+---+---+---+---+---+---+---+---+

|LC |LC |RSP|RSP|LC |LC |LC |LC |     Slot position filling (note 2 left most LC's not drawn)

+---+---+---+---+---+---+---+---+

Now you can see that the 1's are in position 5 and 4, and if you look at the slot numbering of the ASR9006, you can see that these are

the RSP's!! So telnet is delivered to the RSP.

Restrictions for MPP

1) Currently MPP doesn’t keep track of the denied or dropped protocol requests
2) The management protocols need to be enabled explicitly
–MPP configuration doesn’t enable the protocol services.
–MPP is only responsible for making the services available on different interfaces
3) Management requests received on in-band interfaces may not be acknowledged on in-band interfaces
4) RSP Management Ethernet interfaces are by default out-of-band interfaces
5) MPP configuration changes doesn’t affect active sessions established before the changes

6) No MIB Support

Related Information

Cisco Guide to Harden Cisco IOS XR Devices

LPTS Considerations

If you can use only p2p OSPF network type

flow ospf-uc-known rate 0

flow ospf-uc-default rate 0

Note that OSPF p2p network type is the recommended setting even on Ethernet interfaces unless you have multiple routers on the same segment.
Do we really need BGP, LDP-TCP, MSDP, default – for unconfigured sessions

flow bgp-default rate 0

flow ldp-tcp-default rate 0

flow msdp-default rate 0

Xander Thuijs, CCIE #6775

Sr. Tech Lead ASR9000


Comments
Beginner

Hi Xander,

thank you for this useful information (and for all your other contributions ).

I'd have a question about this LPTS point :

I have a network with a configured L3VPN configured the usual way.

A VRF instance is configured in the different routers, and in each router a loopback interface is assigned to that VRF instance.The IP address of the loopback interfaces  is announced by MP-BGP in the MPLS core.

So these IP adresses are reachable (we can ping them from the routers or some workstations). But we can not establish a telnet or ssh session to them. I made some research and found that it is certainly related to the LPTS feature (the command "show lpts ifib statistics" show that everytime I try a telnet/ssh connection, the counter "reject" increments by 1).

I also found that one of MPP requirements is that in case we try to manage a VRF interface, the inband interface we use to join the router must belong to that same VRF, but in the case of L3VPN, inband interfaces are MPLS interfaces, not configured to belong to any VRF instance.

So I believe with MPP configured, there is no way to manage a router through a MPLS interface?

Best regards,

Brahim

Cisco Employee

Hi Brahim! Thank you! And great question!

Check this with standard config:

telnet vrf default ipv4 server max-servers 4

results in :

RP/0/RSP1/CPU0:A9K-BOT#show lpts bindings br | i ,23

Wed Aug 22 15:42:19.160 EDT

0/RSP1/CPU0 TCP  LR IPV4 TCP    default   any          any,23 any

You see where I am going with highlighting this default right?

telnet vrf RED ipv4 server max-servers 4

0/RSP1/CPU0 TCP  LR IPV4 TCP    default   any          any,23 any

0/RSP1/CPU0 TCP  LR IPV4 TCP    RED       any          any,23 any

Now if I add MPP to the mix:

RP/0/RSP1/CPU0:A9K-BOT(config)#control-plane management-plane

RP/0/RSP1/CPU0:A9K-BOT(config-mpp)#inband

RP/0/RSP1/CPU0:A9K-BOT(config-mpp-inband)#interface g0/1/0/0.20

RP/0/RSP1/CPU0:A9K-BOT(config-mpp-inband-if)#allow telnet

RP/0/RSP1/CPU0:A9K-BOT(config-mpp-inband-if)#commit

Now my LPT entries are modified:

0/RSP1/CPU0 TCP  LR IPV4 TCP    default   Mg0/RSP1/CPU0/0 any,23 any

0/RSP1/CPU0 TCP  LR IPV4 TCP    RED       Gi0/1/0/0.20 any,23 any

So without the telnet SERVER for that vrf, the holes are not even pierced to start with.

RP/0/RSP1/CPU0:A9K-BOT(config)#no telnet vrf RED ipv4 server max-servers 4

RP/0/RSP1/CPU0:A9K-BOT(config)#commit

0/RSP1/CPU0 TCP  LR IPV4 TCP    default   Mg0/RSP1/CPU0/0 any,23 any

cheers!

xander

Beginner

Thank you for your answer Xander!

I assume that in your example the interface g0/1/0/0.20 is assigned to vrf RED?

If it is the case, so I think your example and mine are different. I am inserting a diagram for clarification :

LPTS-MPP.JPG

The 4 routers are forming a MPLS backbone ( inter-router links are MPLS links).   I try for example to establish a management session (telnet or ssh) to R2 from the station on the left ( or from any of the other routers , specifying that I am using the vrf RED to establish that session).

In that case, I see the "reject" counter incrementing in the command "show lpts ifib statistics", and  R2 effectively refuses the connection.

But if I try to establish a management session to R1 from the station (or from R3 , but using the LAN link assigned to VRF RED), behaviour is OK, connexions are established. For me, it seems that connections are rejected because the "inband" interface used to reach the loopback interface, and that same loopback interface are not both assigned to vrf RED (inband interface is a MPLS interface, shared between many services).

telnet/ssh services are correctly activated for correct VRFs as you reminded.

I hope my explanation is enough clear

Best regards,

Brahim

Cisco Employee

Aha thanks for that clarification Brahim! The story I have above still applies, but in this picture there is something else in play that I didn't talk about before.

Assuming the picture is complete, and you don't have any PHY interfaces in the vrf on R2, regardless of your telnet vrf RED server statement, the iFIB may not have been programmed for these vrf's because of SVD (Selective VRF Download).

In that mode, the system tries to intelligently determine if an LC is subscriber or core facing and only in limited fashion download vrfs and its programming to the LC.

To "draw" vrf knowledge to the LC, you can either do one of these 2 things:

RP/0/RSP1/CPU0:A9K-BOT(config)#selective-vrf-download disable (commit and reload lc...)

use show SVD [role | status] to track

or you can do what I did to configure any interface on that linecard in that vrf RED which will make the LC pull all vrf RED knowledge.

So to verify check are:

1) check SVD applicability (42+ releases)

2) show lpts bindings brief | i ,23 to see if LPTS holes are pierced and for that vrf

3) make sure that if MPP is configured, interfaces in that VRF are enabled

     Knowing that you may have to create a bogus phy interface as the loopback may not do it on its own)

4) and obviously make sure your telnet server is running in that vrf also.

regards

xander

Beginner

This is becoming more and more interesting..

well I checked those constraints :

      • I am running a 4.1.2 release so for the moment I can not disable SVD. I'll check when I can.

      • But in my topology R1 and R3 already have their "LAN" interface  (assigned to VRF) and their MPLS backbone interface located in the same card (see topology below). And behaviour is the same when I try to manage R3 from the host on the left    (trafic of course flowing through R1... If I try to manage R3 from the LAN, directly to VRF interface, I do not have the problem: telnet/ssh sessions correctly establish)
              • And the same if I try to establish the same remote session from R1 : connections are dropped.

            • I looked for a debug, I used a "debug lpts packet drops" , which gives me this kind of message each time a connection is dropped :

netio[310]: lpts ifib [0xddc545b8/66 if 0x000001a0 VRF 0x60000001 IP4 SOURCE_IP_ADDRESS.62854 -> DESTINATION_IP_ADDRESS
.23 TCP S] to local stack for reject, dropping

I really appreciate your interest !!

Thank you for that,

Regards

Brahim

LPTS-MPP_2.JPG

      Cisco Employee

      Brahim, no prob, if you have 412 then tere is no svd, so we can skip that check.

      if you can telnet from your client to R3 directly, then your telnet server is running in the right vrf.

      since netio sees the packet it means it got punted.

      the problem could either be in ( i )fib, netio or lpts and or could be related to the forwarding from R1 to R3 (label imposition etc).

      One important piece is the nodeID infront of netio (lc or rp and which lc) that might be importnat here too.

      Can you share your show run | i telnet and sh run control-plane.

      And potentially proc restart the 3 procs referenced above and see if that helps.

      if the config comes out clean, I may have to advice you to open a tac case to investigate further.

      thanks

      xander

      Beginner

      Hi Xander,

      I share with you the configuration part you asked for (the vrf RED is the one concerned here).

      I believe loopbak interface doesn't need to be specified in the control-plane. It has been added for testing purposes about this issue

      for the process restarting I'll try to do it when I can, I believe it may be disruptive

      Regards,

      Brahim

      R3#sh run | in telnet

      Building configuration...

      telnet vrf default ipv4 server max-servers 50

      telnet vrf BLUE ipv4 server max-servers 50

      telnet vrf RED ipv4 server max-servers 50

      R3#sh run control-plane

      control-plane

      management-plane

        inband

         interface Loopback1

          allow all

         !

         interface Bundle-Ether1

          allow all

         !

         interface Bundle-Ether2

          allow all

         !

         interface GigabitEthernet0/2/0/19

          allow all

         !

        !

        out-of-band

         vrf BLUE

         interface MgmtEth0/RSP0/CPU0/0

          allow all

         !

        !

      Regards,

      Brahim

      Beginner

      i have read this document. I would like to share my doubts with you.

      In every examples they have mentioned ipv6 address on "out of band" and

      ip adress on "in band" interfaces. Can't we configure ip address on "out of band" and

      ipv6 adress on inband interfaces of ASR 9000. Apart from LPTS and Management plane

      protection, is there any other methods(by using ACL/Routemap/policy-map)

      available in IOS XR for protecting CPU and management plane??

      Cisco Employee

      Just an example...

      RP/0/RSP0/CPU0:A9K-BNG(config)#control-plane

      RP/0/RSP0/CPU0:A9K-BNG(config-ctrl)#management-plane

      RP/0/RSP0/CPU0:A9K-BNG(config-mpp)#inband

      RP/0/RSP0/CPU0:A9K-BNG(config-mpp-inband)#interface all

      RP/0/RSP0/CPU0:A9K-BNG(config-mpp-inband-all)#allow snMP peer address ?

        ipv4  Configure peer IPv4 address on this interface

        ipv6  Configure peer IPv6 address on this interface

      MPP is hardware accelerated security and easier to manage. You can apply access-groups to the service

      (snmp/vty etc) but those packets have then already been processed and handed over to the application where they are denied, so waste of processing cycles. MPP drops them in the hardware as they come in.

      xander

      Beginner

      Hi Alexander,

                    Thanks for your valid information. I would like to share one another doubt.In ASR 9000, can we filter packets based on mac and ip-v6 to protect control and management plane by using LPTS ? Can we use rate-limit of

      queues to limit traffic in ASR also ?? Does LPTS protect both control and management plane or control-plane only?

      Is there any other method available for protecting management-plane?

      Cisco Employee

      Inside XR it is all done by lpts who manages flows, polices them and makes sure they are expected. there is not much to be configured about it. some folks prefer to tune lpts policers, but there is generally no need for it.

      filtering based on mac can be done as part of infrastructure ACL's on the interface.

      But if you have functionality like arp inspection or dhcp snooping then there is little need for that either.

      xander

      Beginner

      Hi Alexander,

                       Thanks for your quick response. In Management plane protection we are configuring "vrf vrf-name" only on out-of-band interfaces. Is this for controlling remote management of the router ?? Is this mandatory ? I am expecting your valid response.

      Beginner

      Hi Xander,

                          I have read this document. But as per my knowlede, there is no " infrastructure ACL's" for MAC filtering available in ASR 9k. Let me know if my knowledge is wrong.

      Cisco Employee

      In fact there is Rakesh! Lucky you

      it's defined like this:

      ethernet-services access-list arpdeny

      10 deny any any 806

      20 permit any any

      xander

      Beginner

      I am trying to allow inband access to the loopback only and created the following configuration

      control-plane

      management-plane

        inband

         interface Loopback0

          allow SSH peer

           address ipv4 10.20.0.21

          !

          allow SNMP

         !

        !

      !

      !

      RP/0/RSP0/CPU0:LAB-B#show lpts bindings brief | i (any.22 )

      Thu Jul 18 06:01:50.248 UTC

      0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Mg0/RSP0/CPU0/0 any,22 any

      0/RSP0/CPU0 TCP  LR IPV4 TCP    default   Lo0          any,22 10.20.0.21

      0/RSP0/CPU0 TCP  LR IPV6 TCP    default   Mg0/RSP0/CPU0/0 any,22 any

      RP/0/RSP0/CPU0:LAB-B#

      so on Lo0 it is listening on the SSH port. However when I try to SSH to the Loopback (from a different ASR Lo0)  I get

      %Error in connect v4 - Connection refused

      These two ASR are connected via a Bundle-Ether and I need to add the Bundle-Ether as well as inband interface to get it to work. However if I do this it also allows me to logon to the interface ip of that Bundle as well.
      can you please explain how this can be prevented.

      CreatePlease to create content
      Content for Community-Ad
      August's Community Spotlight Awards