Feature Manager Troubleshooting On the sup720 when a feature is configured it needs to be determined if this feature is supported by the hardware.  This is accomplished by Feature Manager.  Feature  Manager is a software entity that keeps track of all features that  have  impact on hardware forwarding, and will resolve conflicts as well as share hardware resources among multiple interfaces. If a feature cannot be programmed due to a conflict with another feature OR due to the fact that it is not supported in hardware FM will display this information and generate a message to the syslog. ACL and Netflow TCAM: Throughout  this document ACL TCAM and Netflow TCAM will be referenced.  Each of  these are independent of each other and are stored on different  locations on the PFC. ACL TCAM is the location where features are blocked/allowed or redirected,  based on the configuration of the switch.  This is stored on the PFC and  has a finite number of available entires for information to be stored.   You can read about ACL TCAM on the 6500 more in depth via the following  link: http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a00800c9470.shtml Neflow TCAM is the location where features such as NDE/NAT/WCCP program their forwarding  information.  These features need to punt the first packet of every flow  in order to program entires into Netflow TCAM.   This will hold the  adjancey for the flow installed in order to forward this traffic in  hardware. FM and ACL merge: FM is needed because there is only one ACL lookup per-direction on the 6500.  If   traffic needs to be processed by two different features, that need to   take different paths in hardware, the traffic will need to be software   switched to accomplish this task. An example of m erging two ACLs based features: RACL (Routed ACL) and PBR (Policy-Based Routing), the logic is: – If RACL ACE drops the packet then PBR is not concerned, i.e. merged action is DROP – If RACL ACE permits the packet and PBR ACE doesn’t match this packet then merged action is PERMIT – If RACL ACE permits the packet and PBR ACE does match this packet then merged action is REDIRECT (for PBR) Below is a chart explaining further explaning this logic: Above you can see the merge result when a RACL and PBR match the same traffic. Common FM issues: When troubleshooting FM, there are two common issues that are typically seen: Feature conflicts: certain features impose conflicting requirements on hardware and cannot coexist in HW for example: NAT & PBR. Merge failure of two particular ACLs created by a feature are proportional to the size of each ACL.  This may result to an ACL  bigger than TCAM capacity.  This is commonly referred to as a merge explosion. Below I will go through how to look at ACL TCAM programmed on a layer 3 interface as well as how to find and resolve a feature conflisct  as well as TCAM overutilization. Looking at ACL TCAM when a Feature is configured: When a feature is configured on an interface the sup720 may need to adjust the ACL TCAM on the interface in order to satisfy the requirement for that feature. We can view how ACL TCAM is currently programmed with the “show tcam interface” command, as seen below: 6500-2#show run interface vlan 10 Building configuration... Current configuration : 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 end 6500-2#show tcam interface vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     permit       ip any any Above  we can see that all traffic is being forward in hardware due to the  "permit ip any any" programmed. This works by the same logic as an ACL, where this will permit all traffic passing through this interface.   Since we have not features programmed on this interface on ingress this is expected. We can also see that since we have no features programmed on egress that nothing will be seen in ACL TCAM: 6500-2#show tcam interface vlan 10 acl out ip Now if we applied the following access-list with a log keyword on input we will see the following: 6500-2#sh access-list 1 Standard IP access list 1     10 permit 10.10.10.20 log     20 permit any 6500-2#sh run int vlan 10 Building configuration... Current configuration: 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip access-group 1 in end 6500-2#sh tcam interface vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     punt         ip host 10.10.10.20 any     permit       ip any any Now we can see a punt installed for any traffic from host 10.10.10.20, due to the log keyword on input.  The punt represents traffic that needs to be sent to the RP CPU.    In this case, all this traffic that is sources from 10.10.10.20 will be sent to the RP CPU to be processed switched. Keep in mind that this is only configured on ingress. Under normal operation we should not see any punt adjacencies installed in ACL TCAM on the interface.  A punt implies that the operation cannot be performed in hardware and must be punted to the RP CPU in order to be software switched. **Note   that there are times when you will see a punt in ACL TCAM, but a Netflow entry has been installed overrriding this punt.   If traffic matches both an entry in ACL TCAM and a Netflow entry, the Netflow entry takes precendence. This may  occur with netflow assisted features which need to punt the first  packet of  every flow to create a netflow entry If we configured a deny statement for this IP on ACL 1 we would see a   "deny ip any any" installed irrepsective to the log keyword.  This can  be seen below: 6500-2#sh access-list 1 Standard IP access list 1      5  deny   10.10.10.0 0.0.0.255     10 permit 10.10.10.20 log     20 permit any 6500-2#sh run int vlan 10 Building configuration... Current configuration: 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip access-group 1 in end 6500-2#sh tcam interface vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     deny         ip host 10.10.10.0 0.0.0.255 any     permit       ip any any Note that only a deny statement was installed.  This is due to the fact  there is no reason to install duplicate entries in hardware.  The deny  for the network of 10.10.10.0/24 also encompasses the permit statement  for the 10.10.10.20 entry, hence the log statement is not installed in  hardware to save space. Looking at a feature configured on an interface: Lets take a look at the same interface and see what FM (Feature Manager) says about the features configured on this interface: 6500-2#sh access-list 1 Standard IP access list 1     10 permit 10.10.10.20 log     20 permit any 6500-2#sh run int vlan 10 Building configuration... Current configuration : 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip access-group 1 in end Lets take a look at FM via FIE (Feature Interaction engine) with an ACL configured.  FIE is the component of Feature Manager which implements the feature onto an interface. 6500-2#sh fm fie interface vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : RACL   - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = RACL  +-------------------------------------+             Action Merge Table +-------------------------------------+    RACL               RSLT              R_RSLT            COL +-------------------------------------+    L2R              L2R               P                 0               SB               HB                P                 0               HB               HB                P                 0               L3D              L3D               L3D               0               P                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 2   num# of free TCAM entries in Bank2 = 32758 Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB           HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32757 Interface Vl10 [Egress]: No Features Configured No IP Guardian Feature Configured No IPv6 Guardian Feature Configured No QoS Feature Configured Above we can see that the feature “RACL” (Routed ACL) is configured as ingress feature. We can see that this has invoked FM label 52 to be applied to this interface.  The   FM label is dynamically selected and used as a pointer.  This is done   so that the same label can be applied to multiple interfaces.  This  allows the 6500 to only have to determine the merge required for the same features one time.  We can also see the IP Guardian has not been triggered.  IP Guardian is a portion of feature manager that determines what flowmask is needed on an interface for the features configured.  This is used when netflow assistance is needed for a feature.  It will also protect from having incompatible features programmed into hardware. Some features require specific flowmasks in order to be put into hardware. This can be seen at the following link: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/netflow.html#wp1132875 If no flowmask is configured, the 6500 will dynamically choose the appropriate flowmask via IP Guardian.  If a flow mask is configured using the "mls ip flow <mask>" command, the 6500 will only use this mask for features.  You can look at the specific label with command "show fm label <label #>", which will show us where this label is applied. 6500-2#sh fm label 52 Label 52:   Hardware state is Not Reduced   Force merge is FALSE   Protocol number 0:      Protocol switching is enabled      Configured features:         FM_GUARDIAN (ingress)         IPV4 Default Result Feature (ingress)   Protocol number 2:      Protocol switching is enabled      Configured features:         OTHER Default Result Feature (ingress)   Interfaces (I/E = Ingress/Egress; * = associate pending)      I     Vlan10 In this case we can see that this label is only applied to interface Vlan 10. Troubleshooting a Feature Conflict: A  common issue with FM is having a netflow based feature and an ACL based feature matching the same traffic, configured on the same interface.  Keep in mind that not all ACL/Netflow combinations will cause an issue, such as a security ACL and NAT.  However this would be an issue with NAT and PBR, due to the fact that PBR needs to redirect this traffc on a different path. The problem is traffic cannot be sent on two different paths in hardware, as the sup720 only makes one pass through ACL TCAM per-direction(ingress/egress).  We can not send the traffic down one path for netflow creation and one for the dection made by the ACL otherwise duplicate traffic would occur.   The  following is an example how this would look if you had NAT (netflow  assisted) and PBR (ACL based) configured to match the same traffic on the same interface. Below  we can see that NAT and PBR are matching the same set of traffic.   We know that this will not work due to the fact that the traffic can not  both be punted to create a netflow entry AND be redirected to specific  hop via PBR. ip nat pool test 10.10.101.3 10.10.101.4 prefix-length 24 ip nat inside source list 7 pool test access-list 7 permit 10.10.10.0 0.0.0.255 route-map PBR permit 10 match ip address 1 set ip next-hop 10.10.101.1 ! access-list 1 permit 10.10.10.0 0.0.0.255 ! ip route 0.0.0.0 0.0.0.0 10.10.101.10 ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip nat inside ip policy route-map PBR ! interface Vlan20 ip address 10.10.101.2 255.255.255.0 ip nat outside end Lets take a look at FM on SVI 10 to see how this would be displayed when both PBR and NAT  are applied: 6500-2#show fm fie interface vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : PBR   - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank1 = PBR  +-------------------------------------+             Action Merge Table +-------------------------------------+    PBR                RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               AdR              AdR               P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 12   num# of free TCAM entries in Bank1 = Unknown   num# of merged VMRs in bank 2 = 0   num# of free TCAM entries in Bank2 = Unknown Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32760 Interface Vl10 [Egress]: No Features Configured IP Guardian Feature Configured Current IP Guardian flowmask : Intf Full Flow FIE guardian interaction : Done FIE guardian interaction result : SUCCESS FIE guardian flowmask conflict status : FIE_FLOWMASK_NO_CONFLICT num# of features that bump this interface : 1 features that bump this interface : NAT No IPv6 Guardian Feature Configured No QoS Feature Configured Above  we can see that FM label of 52 was selected and that PBR is only listed  as a feature programmed in hardware.  If we look further down, we   can see the IP guardian has selected to use the “Interface Full Flow”   flowmask.  This is a requirment for NAT to work correct in hardware.  We can also see that NAT is listed as a cause for the interface to be bumped.   This siginifies that traffic matching this feature needs to be sent to the RP  CPU to either be software switched OR  for a the creation of a netflow entry in  hardware. For the creation of a netflow entry to occur the first packet  in every flow needs to be punted, after which it will match on the  netflow entry. ***Note  netflow always takes precedence to whatever is configured in ACL TCAM.  Typically a “redirect” in hardware will demonstrate this traffic is being pushed to the netflow path.  However, this could also be seen a punt.  Please check for a netflow entry for the traffic if a “punt “ is installed and the feature is a netflow assisted feature.  An example of this is demonstrated below. In certain scenarios FM will inform you that a feature can not be placed into hardware with an error message.  The error message will appear similar to the following: %FM-2-FLOWMASK_CONFLICT: Features configured on interface Vlan10 have conflicting flowmask requirements, traffic may be switched in software Lets also take a look at what the TCAM would look like with these features configured. 6500-2#sh tcam int vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     redirect     ip any any fragments     redirect     ip any any fragments     redirect     ip any any     permit       ip any 224.0.0.0 15.255.255.255 fragments     permit       ip any 224.0.0.0 15.255.255.255 fragments     permit       ip any 224.0.0.0 15.255.255.255     policy-route ip 10.10.10.0 0.0.0.255 any fragments     policy-route ip 10.10.10.0 0.0.0.255 any fragments     policy-route ip 10.10.10.0 0.0.0.255 any     permit       ip any any fragments     permit       ip any any fragments     permit       ip any any We  can see that a redirect is installed for all traffic.  This would cause  all traffic to be sent to the netflow path, including the traffic  matched for PBR.  Though we  see the “policy-route” statement installed  for PBR, this traffic will never be hit in hardware due to the redirect  statement matching all traffic. Now I am going to send a flow from an Ixia from  10.10.10.2 to 10.10.102.10. The redirect installed for NAT will cause this traffic to be redirected to the RP CPU for a netflow entry to be installed.  However,   If we look at netflow we will see the entry installed for this traffic: 6500-2#sh mls netflow ip Displaying Netflow entries in Active Supervisor EARL in module 2 DstIP           SrcIP           Prot:SrcPort:DstPort  Src i/f      :AdjPtr ----------------------------------------------------------------------------- Pkts         Bytes         Age   LastSeen  Attributes --------------------------------------------------- 0.0.0.0         0.0.0.0         0   : 0      : 0        --           :0x0         30           1380          72    13:31:25   L3 - Dynamic 10.10.102.10    10.10.10.2      tcp : 0      : 0     Vl10          :0x0         0            0             82    13:31:25   L3 - Dynamic We can see in red above that no packets/bytes have hit this entry.  This is because this traffic is being software forwarded.  This is due to the fact that PBR is configured for the same subnet (10.10.10.x).  This traffic can not both be redirect to the next-hop via PBR and go through the netflow patch in hardware.  This traffic is pushed to the RP CPU in an attempt to perform both operations.  We can also see that the CPU is high due to this traffic from IXIA: 6500-2#sh processes cpu CPU utilization for five seconds: 57%/57%; one minute: 13%; five minutes: 7% **Note all this will cause the CPU to be drive by interrupts not a process. Notice that as soon as we remove PBR we can see that this entry is now being hit: 6500-2(config)#int vlan 10 6500-2(config-if)#no ip policy route-map PBR 6500-2(config-if)#do sh mls netflow ip Displaying Netflow entries in Active Supervisor EARL in module 2 DstIP           SrcIP         Prot:SrcPort:DstPort  Src i/f      :AdjPtr ---------------------------------------------------------------------------- Pkts         Bytes          Age   LastSeen  Attributes --------------------------------------------------- 10.10.102.10    10.10.10.2      tcp : 0      : 0        Vl10       : 0x80002      77239        3552994       6     13:36:03   L3 - SwInstalled 0.0.0.0         0.0.0.0         0   : 0      : 0        --         :0x0         126          5796          351   13:35:57   L3 – Dynamic We  can also see that the netflow entry has moved from SwInstalled compared  to Dynamic as well as an AdjPtr created for this traffic, which will be  used for any further traffic in this flow. ***Note - When NAT is configured, we need to drive a different  flowmask for fragment packets so that the fragment packets (basically  the first fragments which have L4 port information) are not hardware  shortcut by matching the NAT netflow shortcut (by driving different flow  mask for fragment packets, fragment packets are prevented from matching  the NAT netflow shortcut installed with L4 port information) and these  packets need to be translated in software (first fragment and later  fragments).  We will continue to see high CPU if the traffic is  fragemented.  We have one option supported for PFC3B and later: "mls ip nat netflow-frag-l4-zero" which when configured, the netflow lookup key formed to lookup/match the netflow table would have the L4 information zeroed out for fragment packets (including first fragments) so that the first fragment (which has L4 information) does not match the netflow short installed for NAT in this case. We can also see that the CPU has dropped, as expected: 6500-2#sh proc cpu | ex 0.00 CPU utilization for five seconds: 0%/0%; one minute: 45%; five minutes: 20% Now, lets take a look to see if FM has changed: 6500-2#sh fm fie int vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : [empty] - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 num# of strategies tried : 1   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = Unknown   num# of merged VMRs in bank 2 = 6   num# of free TCAM entries in Bank2 = Unknown Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32746 Interface Vl10 [Egress]: No Features Configured IP Guardian Feature Configured Current IP Guardian flowmask : Intf Full Flow FIE guardian interaction : Done FIE guardian interaction result : SUCCESS FIE guardian flowmask conflict status : FIE_FLOWMASK_NO_CONFLICT num# of features that bump this interface : 1 features that bump this interface : NAT No IPv6 Guardian Feature Configured No QoS Feature Configured Notice  that FM  still shows that NAT is listed as a feature that is bumping the   interface.  This is due to the fact that the first flow for all traffic   that needs to be NATed must be punted to the CPU for the creation of a   netflow entry.  However, as we can see above this netflow entry is now  being used in preference to what is configured in ACL TCAM. Notice   that the flowmask is still the same, due to the fact that an   interface-full flowmask is required for NAT to work in hardware. We   can also see that PBR is no longer listed as an installed feature and   is now listed as “empty” since this feature was removed from this   interface. **Note   that the FM lael has not changed in this case.  This is due to the  fact  that FM assigns these dynamially and no other feature had already   reserved this label.  This is due to the fact that this is the only   interfaces on this 6500 with features configured. If we look at ACL TCAM we can see that only a redirect is now installed for NAT. 6500-2#sh tcam int vlan 10 acl in ip * Global Defaults not shared Entries from Bank 0 Entries from Bank 1     redirect     ip any any fragments     redirect     ip any any fragments     redirect     ip any any     permit       ip any any fragments     permit       ip any any fragments     permit       ip any any Traffic that does not match the redirect will be forwarded normally. Below shows what this would look like if  NAT were removed rather than PBR. The TCAM would look like the following: 6500-2#sh tcam int vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     permit                   ip any 224.0.0.0 15.255.255.255     policy-route          ip 10.10.10.0 0.0.0.255 any     permit                   ip any any If you look at FM we can see that PBR is only listed in hardware and NAT is no longer listed as a feature that is configured: 6500-2#sh fm fie interface vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 6 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : PBR   - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank1 = PBR  +-------------------------------------+     Action Merge Table +-------------------------------------+    PBR        RSLT      R_RSLT    COL +-------------------------------------+    SB       HB        P         0       AdR      AdR       P         0       X        P         P         0    +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 3   num# of free TCAM entries in Bank1 = Unknown   num# of merged VMRs in bank 2 = 0   num# of free TCAM entries in Bank2 = Unknown Slot(s) using the protocol OTHER : 6 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+     Action Merge Table +-------------------------------------+    OTH_DEF    RSLT      R_RSLT    COL +-------------------------------------+    SB       HB        P         0       X        P         P         0    +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32734   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32742 Interface Vl10 [Egress]: No Features Configured No IP Guardian Feature Configured No IPv6 Guardian Feature Configured No QoS Feature Configured TCAM overutilization from a feature configuration: In certain cases the sup720 will not be able to fit all of the required parametes into the TCAM on the switch.  This most commonly an issue with overutilization of the ACL or Netflow TCAM.  When the traffic can not be fit into hardware it is fowarding in software via the RP CPU. If we attempt to install an ACL that is to large to fit into TCAM we will see the following messages.  We can also see via the "show fm summary" command that the interfaces where these ACL's were reduced FM is inactive, which means this traffic is being software switched. If we see an issue like this occur we need to confirm the space that is left with in TCAM.  This can be done via the "show tcam counts" command will which show ACL and QOS TCAM space available: In order to look at this information for a DFC module (which will hold independet TCAM from the supervisor), you need to specify the module after the "show tcam counts" command.  For example, "show tcam counts module <mod#>" **Note that TCAM counts displayed above may be after the TCAM merge has failed and may not show the TCAM has been overutilized. If you have run out of Netflow TCAM, you will see the following error message: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM Utilization [99%] You can check your TCAM utilization with the "show mls netflow table-contention detailed" command. 6500#show mls netflow table-contention detailed Earl in Module 6 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization             :     0% ICAM Utilization             :     0% Netflow TCAM count           :     3 Netflow ICAM count           :     0 Netflow Creation Failures    :     0 Netflow CAM aliases          :     0 The above shows us the Netflow TCAM utilization as a percentage, as well as the number of entires that are currently in TCAM. **Note the ICAM listed above is used when two TCAM entries resolve to the same HASH, the ICAM is used to resolve these common entires by adding an additional parameter to distinguish these flows. If you still need assistance troubleshooting FM, please open a Cisco TAC case to investigate your issue further.  ... View more

Troubleshooting with a NETDR capture on a sup720/6500 A netdr capture is preformed on the MSFC CPU controller.  This is the closest location you can capture a packet on the MSFC in order to determine why traffic is being punted to the SP OR RP CPU on the MSFC.  With a Sup720 or Sup32 it allows one to capture packets on the RP or SP inband. The netdr command can be used to capture both Tx and Rx packets in the software-switching path. Cat6500#debug netdr capture ? acl                     (11) Capture packets matching an acl and-filter              (3) Apply filters in an and function: all must match continuous              (1) Capture packets continuously: cyclic overwrite destination-ip-address  (10) Capture all packets matching ip dst address dstindex                (7) Capture all packets matching destination index ethertype               (8) Capture all packets matching ethertype interface               (4) Capture packets related to this interface or-filter               (3) Apply filters in an or function: only one must match rx                      (2) Capture incoming packets only source-ip-address       (9) Capture all packets matching ip src address srcindex                (6) Capture all packets matching source index tx                      (2) Capture outgoing packets only vlan                    (5) Capture packets matching this vlan number <cr> OPTIONS: ·       Using the continuous option, the switch will capture packets on the RP-inband continuously fill the entire capture buffer (4096 packets) and then start to overwrite the buffer in a FIFO fashion. ·       The tx and rx options will capture packets coming from the MSFC and going to the MSFC respectively. ·       The and-filter and the or-filter specify that an and or an or will be applied respectively to all of the options that follow. For example, if you use the syntax below, then both option #1 and option #2 must match for the packet to be captured. Similarly, if the or-filter is used either option #1 or option #2 or both must match for the packet to be captuered. o    debug netdr and-filter option#1 option#2  ·       The interface option is used to capture packets to or from the specified interface. The interface can be either an SVI or a L3 interface on the switch. ·       The vlan option is used to capture all packets in the specified VLAN. The VLAN specified can also be one of the internal VLANs associated with a L3 interface. ·       The srcindex and dstindex options are used to capture all packets matching the source ltl and destination ltl indices respectively. Note that the interface option above only allows the capture of packets to or from a L3 interface (SVI or physical). Using the srcindex or dstindex options allows the capture of Tx or Rx packets on a given L2 interface. The srcindex and dstindex options work with either L2 or L3 interface indices. ·       The ethertype option allows the capture of all packets matching the specified ethertype. ·       The source-ip-address and destination-ip-address options allow the capture of all packets matching the specified source or destination IP address respectively. Example of Netdr and Interpreting the data: Below is an example of capturing traffic destined to 10.100.101.10 sourced from 10.10.10.2 going to the RP CPU: 6500-2#debug netdr cap rx and-filter source-ip-address 10.10.10.2 destination-ip-address 10.100.101.10 6500-2#sh netdr cap A total of 4096 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets ------- dump of incoming inband packet ------- interface Vl10 , routine mistral_process_rx_packet_inlin, timestamp 00:00:11 dbus info : src_vlan 0xA(10), src_indx 0xC0 (192), len 0x40(64)   bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)   10020400 000A0000 00C00000 40080000 00060468 0E000040 00000000 03800000 mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)   requeue 0, obl_pkt 0, vlan 0xA(10) destmac 00.15.C7.26.FB.80, srcmac 00.00.01.00.06.00, protocol 0800 protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 0   df 0, mf 0, fo 0, ttl 100, src 10.10.10.2, dst 10.100.101.10     tcp src 0, dst 0, seq 0, ack 0, win 0 off 5 checksum 0x265C Red = Ingress Vlan of traffic Blue = Layer 3 interface traffic is coming from Green = Ethertype and SRC/DST MAC addresses Purple = IP Header Orange = SRC index (source of ingress traffic). Dark Red = Dest Index (where traffic is being sent). Aqua = Flood bit You can track down the physical port where this traffic is sourced from by tracking down the mac-address (which is the last device to own this traffic).   This can be done in the following manner: 6500-2#show mac-address address 0000.0100.0600 Legend: * - primary entry         age - seconds since last seen         n/a - not available   vlan   mac address     type    learn     age              ports ------+----------------+--------+-----+----------+-------------------------- Active Supervisor: *   10  0000.0100.0600   dynamic  Yes          0   Gi4/1 You can also look at ingress location of this traffic by taking the source index/ltl index.  The LTL (local target logic) is used by the sup720 forwarding engine in order to distinguish locations to forward a packet.  These indexes could be pointed towards a specific port or to an internal location on the switch. In the example below you will use this index to determine ingress port of the traffic above. Below is an example of how to determine the source of a packet based on the source LTL index of 0xC0 that is seen in the capture above. 6500-2#remote command switch test mcast ltl index c0 index 0xC0 contain ports 4/1 We can see this index is used for port 4/1.  We now know this traffic is coming from port 4/1, which was also confirmed based on the port where the MAC is learned.  Determining the source of this traffic via LTL can be very useful when determining the source of the traffic during bridging loop, if multiple types of traffic are being flooded/sent to the CPU, but sent from the same port.   You can analyze the netdr information and look to see if it is coming from the same source index. You can also determine where the traffic is being sent looking at the destination index.  In this case we can see that the traffic is being sent to the destination index of 0x380, which is the RP CPU:                                                                         6500-2#rem com sw test mcast ltl ind 380 index 0x380 contain ports 2/R 2/R = module/Router 6500-2#show module Mod Ports Card Type                              Model              Serial No. --- ----- -------------------------------------- ------------------ -----------   2    2  Supervisor Engine 720 (Active)         WS-SUP720-BASE     SAD074205FP   4   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL1107GS80 In the above case we can see that module 2 is the supervisor with R indicating this is the RP CPU. If the packet that is captured has the flood bit set in the netdr capture, this traffic is being flooded to all ports within the vlan.  In order to see what ports are included within this index you will need to change the destination index to seen in the netdr to account for this bit being set.  The change that will be performed is to add 8 to the beginning of the destination index (which means the first bit in the LTL is set).   The following is an example of an ARP request broadcast frame, which is flooded to the CPU and to all ports in a vlan.   I will also show how you can interpret this data. 6500-2#sh netdr cap      A total of 1 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets ------- dump of incoming inband packet ------- interface Vl10, routine mistral_process_rx_packet_inlin, timestamp 00:00:03 dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x40(64)   bpdu 0, index_dir 0, flood 1 , dont_lrn 0, dest_indx 0x400A(16394)   38020400 000A0000 00C00000 40080000 E0000060 8E0FFBFC 00000008 400A0000 mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)   requeue 0, obl_pkt 0, vlan 0xA(10) destmac FF.FF.FF.FF.FF.FF, srcmac 00.0C.29.21.DE.6E, protocol 0806 layer 3 data: 00010800 06040001 000C2921 DE6EAC10 3E800000 00000000               AC103E02 FFFF0000 00000000 00000000 00000000 00000000               00000000 00000000 0000FFFF In the above example the destination index is set to 0x400A and the flood bit is set to 1, both are sent to red.   In this case in order to see where this packet would be sent we need to add 8 to first value in the index.  For example, the index 0x400A would be changed  0xC00A, which signifies that the flood bit is set when looking up this index.  Also, If an index of 0xC12 would become 0x8C12, when the index is only 3 Hex values. With the two cases above this packet would be flooded in vlan 10 (A = 10) and vlan 3090 (c12= 3090). The rest of the Index after the flood bit is a reprsentation of the Vlan. We can look at this information from the example netdr above with the following command: 6500-2#remote command switch test mcast ltl index C00A index 0xC00A contain ports 2/R,, 4/1 If we do not adjust this index when the flood bit is set the index will be listed as “empty”, which can be seen below: 6500-2#remote command switch test mcast ltl index 400A index 0x400A contain ports * empty * If you need still assistance with interpreting this information, please open Cisco TAC case. ... View more

Troubleshooting high CPU on a 6500 with sup720 The purpose of this document is to cover how to determine the cause of high CPU on a 6500/7600 with a sup720.  The troubleshooting methods discussed in this documentation will make it possible to determine the cause of 90% of all high CPU issues on the sup720. The majority of high CPU on the sup720 is related to CPU usage on the MSFC, thus the majority of this document will cover high CPU on the MSFC. Because it would not be possible to cover every possible reason high CPU can be caused on the sup720, I will demonstrate how to use some of the tools built-in to the sup720 to show general methods on how to narrow down the cause of high CPU. If you are unable to determine the reason based on this documentation, please open a TAC case to investigate this issue further.   **Note that these methods can be used to determine the cause of high CPU on a RSP720, Sup32 and VS-S720, due to common architecture.     Determining Where the CPU utilization is occurring:     Within the sup720 6500, there are two types of CPU’s.   One is used for layer 2 operations and is commonly referred as the SP (Switch Processor) CPU.  The other CPU is used for layer3/4 operations and it commonly referred to as the RP(Route Processor) CPU.  Both of these processors are located on the MSFC3 complex, each with a 1 gig in-band channel to the supervisor. Also depending on the module you may also have a DFC (Distributed Feature Card) to perform forwarding locally on that module.  The DFC also has its own CPU, which performs processing locally on the line card.  Under certain scenarios high CPU can be seen on these modules.   High CPU on the SP (Switch processor): High CPU on the SP is much less common than high CPU on the RP.   The reasons for high CPU on the SP are typically related to layer 2 operations of the sup720, such a spanning-tree (processing of BPDU's) or processing IGMP snooping/IGMP queries/membership reports as well as LACP/PAGP.     You can view the CPU utilization using the following command:     SP CPU Util:   Switch# remote command switch show process cpu   OR   Switch#remove login switch Switch-sp#show process cpu     High CPU on the RP (Route Processor): This will be traffic that needs to be processed for layer 3 operations, such as ARP, HSRP, forwarding traffic in software.   Below I will go over troubleshooting steps when seeing high CPU on the IP Input/ARP input process as well as CPU utilization caused by interrupt switched traffic on the RP CPU.     You can view the CPU utilization using the following command:   RP CPU Util: Switch#show process cpu   High CPU on a DFC/module: The CPU on the DFC will help in programming TCAM and router in hardware, since each DFC has its own TCAM. High CPU on a DFC is not very common and can occur for a few different reasons.  One reason you may see high CPU on the DFC is due to Netflow Data Export.   Typically CPU from NDE is expected, but in rare instances it can become high enough to disrupt other processes. You can view the CPU utilization using the following command: DFC CPU Util:   Switch# attach <module> Switch-DFC#show process cpu   Types of CPU utilization:   There are two type of CPU utilization within IoS, interrupt and process.   Process based CPU utilization:   CPU utilization caused by a process can be caused by few reasons listed below:   1.) Processes switched traffic.  This is traffic that is hitting a specific process in order to be forwarded OR processed by the CPU.  An example of each would traffic being forwarded via the "IP Input" process OR control-plane traffic hitting the "PIM process".   2.) A process trying to clean up tables/previous actions performed.  This can be seen in process such a "CEF Scanner" OR "BGP Scanner", which are used to clean/update the CEF and BGP tables.   Interrupt based CPU utilization:   CPU caused by an interrupt is always traffic based.  Interrupt switched traffic, is traffic that does not match a specific process, but still needs to be forwarded.   Determining the type of CPU utilization: Process and Interrupt CPU utilization are listed within the "show process cpu" command.   This is broken down below on how to determine what percentage of the CPU utilization is due to interrupt traffic or processed switched traffic:   6500-3#sh proc cpu CPU utilization for five seconds: 0% / 0% ; one minute: 0%; five minutes: 0%     Red - Percentage of total CPU utilization Blue - Percentage of the CPU that is caused by Interrupts.     Percentage of process CPU util. = Total CPU - Interrupt CPU util. Common reasons for HIGH CPU on the MSFC/RP:   IP traffic with a TTL of 1 - Due to the fact we need to send an IP unreachable message to the host letting them know the message has expired in transit.  This cannot be done in hardware and thus the packet must be punted to the MSFC.  Find the device sending traffic the TTL of 1 and stop is from sending traffic, increase the TTL OR install the MLS TTL rate-limiter. Using an ACL with the log keyword - Since a log keyword requires a syslog message to be generated this must be punted to the RP CPU as it cannot be done in hardware. Remove the log keyword from the ACL. Using a PBR route-map without a set statement - Any traffic that matches a PBR route-map with no set statement will be punted.  This is due to the fact that we need to program the next-hop in hardware and if the next-hop is not known, this traffic must be punted to determine the next hop.  Configure a set statement OR remove the policy route from the interface. FIB TCAM Exception - If you try to install more routes than are possible into the FIB TCAM  you will see the following error message in the logs:     CFIB-SP-STBY-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched   %CFIB-SP-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched %CFIB-SP-STBY-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched   This error message is received when the amount of available space in the TCAM is exceeded. This results in high CPU. This is a FIB TCAM limitation. Once TCAM is full, a flag will be set and FIB TCAM exception is received. This stops from adding new routes to the TCAM. Therefore, everything will be software switched. The removal of routes does not help resume hardware switching. Once the TCAM enters the exception state, the system must be reloaded to get out of that state. You can view if you have hit a FIB TCAM exception with the following command: 6500-2#sh mls cef exception status Current IPv4 FIB exception state = TRUE Current IPv6 FIB exception state = FALSE Current MPLS FIB exception state = FALSE When the exception state is TRUE, the FIB TCAM has hit an exception. The maximum routes that can be installed in TCAM is increased by the mls cef maximum-routes command.   This issue is common when trying to route a full BGP table on PFC-3A or a PFC-3B. **Note a failover of the supervisors in dual supervisor system will not recover this exception, even through the “show mls cef exception status” will no longer indicate a FIB exception.  A full reload of the switch is required. ICMP redirects - If traffic is taking a path that is not efficient, an ICMP redirect will be sent out to inform the host of a better next-hop.  This will cause the packet to be punted in order to trigger the MSFC to send the ICMP redirect to the host. This can be seen when performing a netdr capture.  An example of using netdr can be seen in the “Tools used to determine the source of the CPU utilization:” section Turn off icmp redirects to stop this traffic from being punted.  However this is an indication of network inefficiency that was attempting to be dynamically resolved.   User interaction is needed in order to track down this inefficiency.  If you need assistance in determining why ICMP redirects are being generated please open a TAC case   CEF Glean adjacency  - This can happen when no ARP resolution for the next hop.  All traffic must be punted in order to trigger an ARP request for the next hop.  This will always manifest it self as interrupt based traffic. To protect the RP CPU from this issue you can implement the Glean adj. mls rate-limiter. Netflow and ACL feature configured on the same interface matching the same traffic - You cannot have an ACL based feature and a Flow based feature configured on the same interface for the same traffic.  An Example of this would be having NAT and PBR configured on the same interface matching the same traffic. NAT is netflow assisted, as the first packet in every flow would need to be punted to create the netflow entry in hardware.  Once the netflow entry is created all subsequent packets will hit this hardware netflow entry and thus be forwarded in hardware. Policy-Based Routing is ACL based. This will create a “policy-route” state when a route-map is configured to use PBR and applied to that interface.   This will point to a special adjacency, which is where the next-hop is specified in the “set” statement of the route-map. The issue comes when a packet matches both the NAT and the PBR feature, the traffic can not be sent to the CPU to be put into Netflow AND be redirected to the PBR special adj, thus this traffic must be software switched.  If these two features overlap, these features are taken out of hardware and the traffic is software switched.  When this occurs neither feature may be applied to the matching traffic. If a packet does not match both the ACL based feature and Netflow based feature match criteria then the relevant function (ACL based or Netflow based) will be performed in hardware. Therefore, for proper hardware based performance in situations where ACL based features and Netflow based features are configured on the same interfaces it is important to have unique policies.   To work around this problem do not have both an ACL based and Netflow based feature configured on the same interface, matching the same traffic.   You can read more about troubleshooting feature conflict issues via the following link:   https://supportforums.cisco.com/docs/DOC-15670   Directed Broadcast traffic – All broadcast traffic must be sent to the MSFC on a vlan when a layer 3 interface is configured within that vlan.  This includes directed broadcast traffic.   Use multicast instead of directed broadcast.     Bridging loop - If a bridging loop occurs on the network, this could cause high CPU on the MSFC. All broadcast traffic must be sent to the MSFC on a vlan when a layer 3 interface is configured within that vlan.     You can determine what traffic is hitting the CPU by using a netdr capture to track down the source interface of the loop (See Using Netdr to determine traffic punted to the CPU section).      GRE with non-unique  tunnel source - On the sup720, tunnel sources must be unique for all tunnels. Tunnels with a non-unique source will be software switched. The workaround for this limitation is to use either unique loopback interfaces for every GRE tunnel OR  use secondary addresses on a loopback interface for the tunnel source addresses.       For more information see CSCdy72539. You may also see the following error:   %Warning: Using same source IP for more than one IP/GRE tunnels may cause software switching packets for tunnels using this address. If possible, use a unique tunnel source for Interface Tunnel <tun#>                  Other common unsupported features on Sup720-PFC3:     The following features/traffic types are common features that are not supported by the 6500 and will cause high CPU if implemented:     **Note this is not an exhausted list and there may be unsupported features not listed below.     NBAR Traffic with IP options field set. Multicast RPF drops RSVP (INTSERV QOS) *can be used for tunnels CEF accounting Multicast traffic and NAT – see CSCek78254     The following link will give a larger list of all unsupported features and commands on the sup720-PFC3:     http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/features.html#wp3691673     Tools used to determine the source of the CPU utilization:       Determine the source of RP CPU utilization using interface buffers:     **Note** you will only be able to see traffic in the interface buffers on a layer 3 interface if the traffic is being processed switched (see “Determining type of CPU utilization” above).  This will not work when traffic is being interrupt switched.  In the case of interrupt switched traffic use the netdr capture instead.     One of the quickest ways to determine the layer 3 interface that is the source of traffic that is causing high CPU is to see which interface has a large amount of drops flushes on the interfaces input queue.  The input queue on a layer 3 interface is the CPU queue for that interface on the sup720.   If we ever see packets/drops on the input queue on the sup720 it is always due to traffic that is being sent towards the CPU.  You can narrow down the location of such an interface with the following commands:     6500-2#show interface | include  is up|drop   Vlan10 is up, line protocol is up   Input queue: 74/75/18063/18063 (size/max/drops/flushes); Total output drops: 0 Vlan20 is up, line protocol is up Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0   We can see that SVI (Switched Virtual Interface) 10 has 74 packets in its buffer, whose queue size is 75 packets. This demonstrates that a large amount of traffic is being punted on this interface to the RP CPU, since this queue is full.     Now that we can see a large amount of traffic within this queue, we can look at what is in this queue with the command "show buffers input-interface vlan 10 header".  This command will display the IP header of the packet so we can attempt to determine the source.  If you want to look at the entire packet you can use the command  "show buffers input-interface vlan 10 packet".     Below is the output from this command for SVI 10     6500-2#sh buffers input-interface vlan 10 header     Buffer information for Small buffer at 0x4667A08C   data_area 0x802F664, refcount 1, next 0x466AE968, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x802F6DA, datagramsize 60, maximum size 308   mac_start 0x802F6DA, addr_start 0x802F6DA, info_start 0x0   network_start 0x802F6E8, transport_start 0x802F6FC, caller_pc 0x41F78790       source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,     TOS: 0 prot: 6, source port 0, destination port 0     Above we can see the basic information about this traffic that is included in the IP header, including the TOS, TTL and protocol encapsulated within the IP header.     If we viewed the entire packet we can look at more in depth information including the layer 2 information, as can be seen below:     6500-2#sh buffers input-interface vlan 10 packet     Buffer information for Small buffer at 0x466A23B0   data_area 0x80340A4, refcount 1, next 0x466E991C, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x52836BE4 (Vlan10), if_output 0x0 (None)    inputtime 16:32:10.292 (elapsed 00:00:50.608)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x803411A, datagramsize 60, maximum size 308   mac_start 0x803411A, addr_start 0x803411A, info_start 0x0   network_start 0x8034128, transport_start 0x0, caller_pc 0x41F78790       source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,     TOS: 0 prot: 6, source port 0, destination port 0            0: 0015C726 FB80 0000 01000600 0800 4500  ..G&{.........E.       16: 002E0000 00000106 3651 0A0A 0A02 0A64   ........6Q.....d       32: 650A 0000 00000000 00000000 00005000  e.............P.       48: 0000265C 00000001 02030405 FD        ..&\........}             Red = Dest MAC Blue = Source MAC Green = Ethertype (0x800 for IP traffic) Purple = Src. IP Orange = Dest IP   Using “show ip traffic” statistics to see why traffic is punted:   6500-2#show interface | i is up|drop   Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Vlan10 is up, line protocol is up   Input queue: 74/75/18063/18063 (size/max/drops/flushes); Total output drops: 0 Vlan20 is up, line protocol is up   SVI 10/Interface Vlan 10 is receiving a large amount of traffic that is being punted to the RP CPU.  When we look at what is in this queue with the command "show buffers input-interface vlan 10 header".   Below is the output from this command for SVI 10     6500-2#sh buffers input-interface vlan 10 header     Buffer information for Small buffer at 0x4667A08C   data_area 0x802F664, refcount 1, next 0x466AE968, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x802F6DA, datagramsize 60, maximum size 308   mac_start 0x802F6DA, addr_start 0x802F6DA, info_start 0x0   network_start 0x802F6E8, transport_start 0x802F6FC, caller_pc 0x41F78790     source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,   TOS: 0 prot: 6, source port 0, destination port 0     Buffer information for Small buffer at 0x4667C7E8   data_area 0x80314A4, refcount 1, next 0x46695FD0, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x803151A, datagramsize 60, maximum size 308   mac_start 0x803151A, addr_start 0x803151A, info_start 0x0   network_start 0x8031528, transport_start 0x803153C, caller_pc 0x41F78790     source: 10.10.10.1, destination: 10.10.10.2, id: 0xD096, ttl: 255, prot: 1   Since at this point we are unsure why this traffic is being punted, we can look at “show ip traffic" statistics to see why this traffic is being punted to the CPU.  First start by clearing the IP traffic statistics.  We can then see what is incrementing in these counters to see what would be the cause:   6500-2#clear ip traffic Clear "show ip traffic" counters [confirm] 6500-2#sh ip traffic IP statistics:   Rcvd:  33516 total, 0 local destination          0 format errors, 0 checksum errors, 33516 bad hop count <------We can see that the bad Hop count in this case is incrementing          0 unknown protocol, 0 not a gateway          0 security failures, 0 bad options, 0 with options   Opts:  0 end, 0 nop, 0 basic security, 0 loose source route          0 timestamp, 0 extended security, 0 record route          0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump          0 other     Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 couldn't fragment   Bcast: 0 received, 0 sent   Mcast: 0 received, 0 sent   Sent:  0 generated, 0 forwarded   Drop:  40005 encapsulation failed, 0 unresolved, 0 no adjacency          0 no route, 0 unicast RPF, 0 forced drop          0 options denied, 0 source IP address zero     ICMP statistics:   Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable         0 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench         0 parameter, 0 timestamp, 0 info request, 0 other         0 irdp solicitations, 0 irdp advertisements         0 time exceeded, 0 timestamp replies, 0 info replies   Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply         0 mask requests, 0 mask replies, 0 quench, 0 timestamp         0 info reply, 58464 time exceeded, 0 parameter problem   <---- We can also the 6500 is sending ICMP TTL expired messages as well.         0 irdp solicitations, 0 irdp advertisements   <snip>   Looking at the traffic statistics we can see that the bad hop count counter is incrementing and the switch is sending ICMP time exceeded messages.   On the 6500 all traffic with a TTL of 1 is punted to the CPU so that an ICMP TTL expired message can be sent to the host who sent this traffic.   Also, the first packet in the buffer can be seen to have TTL of 1, which is why this traffic is punted.  We can see that the 2nd packet is sourced from 10.10.10.1 (SVI 10) sent to 10.10.10.2. This packet is an ICMP TTL expired message.     Using Netdr to determine traffic punted to the CPU:     A netdr capture is preformed on the MSFC CPU controller.  This is the closest location you can capture a packet on the MSFC in order to determine why traffic is being punted to the MSFC/RP CPU.  With a Sup720 or Sup32 it allows one to capture packets on the RP or SP inband. The netdr command can be used to capture both Tx and Rx packets in the software-switching path.     Cat6500#debug netdr capture ? acl                     (11) Capture packets matching an acl and-filter              (3) Apply filters in an and function: all must match continuous              (1) Capture packets continuously: cyclic overwrite destination-ip-address  (10) Capture all packets matching ip dst address dstindex                (7) Capture all packets matching destination index ethertype               (8) Capture all packets matching ethertype interface               (4) Capture packets related to this interface or-filter               (3) Apply filters in an or function: only one must match rx                      (2) Capture incoming packets only source-ip-address       (9) Capture all packets matching ip src address srcindex                (6) Capture all packets matching source index tx                      (2) Capture outgoing packets only vlan                    (5) Capture packets matching this vlan number <cr>       OPTIONS: Using the continuous option, the switch will capture packets on the RP-inband continuously fill the entire capture buffer (4096 packets) and then start to overwrite the buffer in a FIFO fashion. The tx and rx options will capture packets coming from the MSFC and going to the MSFC respectivey. The and-filter and the or-filter specify that an and or an or will be applied respectively to all of the options that follow. For example, if you use the syntax below, then both option #1 and option #2 must match for the packet to be captured. Similarly, if the or-filter is used either option #1 or option #2 or both must match for the packet to be captuered. debug netdr and-filter option#1 option#2 The interface option is used to capture packets to or from the specified interface. The interface can be either an SVI or a L3 interface on the switch. The vlan option is used to capture all packets in the specified VLAN. The VLAN specified can also be one of the internal VLANs associated with a L3 interface. The srcindex and dstindex options are used to capture all packets matching the source ltl and destination ltl indices respectively. Note that the interface option above only allows the capture of packets to or from a L3 interface (SVI or physical). Using the srcindex or dstindex options allows the capture of Tx or Rx packets on a given L2 interface. The srcindex and dstindex options work with either L2 or L3 interface indices. The ethertype option allows the capture of all packets matching the specified ethertype. The source-ip-address and destination-ip-address options allow the capture of all packets matching the specified source or destination IP address respectively.   Below is an example of capturing traffic destined to 10.100.101.10 sourced from 10.10.10.2 going to the RP CPU:     6500-2#debug netdr cap rx and-filter source-ip-address 10.10.10.2 destination-ip-address 10.100.101.10   6500-2#sh netdr cap   A total of 4096 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets   ------- dump of incoming inband packet ------- interface Vl10 , routine mistral_process_rx_packet_inlin, timestamp 00:00:11 dbus info : src_vlan 0xA(10), src_indx 0xC0 (192), len 0x40(64) bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896) 10020400 000A0000 00C00000 40080000 00060468 0E000040 00000000 03800000 mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)   requeue 0, obl_pkt 0, vlan 0xA(10) destmac 00.15.C7.26.FB.80, srcmac 00.00.01.00.06.00, protocol 0800 protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 0   df 0, mf 0, fo 0, ttl 100, src 10.10.10.2, dst 10.100.101.10     tcp src 0, dst 0, seq 0, ack 0, win 0 off 5 checksum 0x265C       Red = Ingress Vlan of traffic   Blue = Layer 3 interface traffic is coming from   Green = Ethertype and SRC/DST MAC addresses   Purple = IP Header   Orange = SRC index (source of ingress traffic).   Dark Red = Dest Index (where traffic is being sent).       You can use this information to track down the source of the traffic being punted.  Please refer to Troubleshooting with a NETDR capture on a sup720/6500 documention for a further explanation of how to interpret this data.   Please open a TAC case if you need further assistance interpreting this data.     Using a CPU SPAN to determine traffic being punted to the CPU     This capture is performed on the ASIC, which is connected to the RP/SP CPU.    This will allow you to replicate traffic that is being sent to the RP or SP CPU to a capture device.   This can be handy for determine the cause of the HIGH CPU OR determining if traffic is being sent to or from the CPU for processing (such as HSRP/OSPF/PIM control plane traffic).       When using the 12.2(18)SXF train and earlier the  configuration for an inband span session is as follows:     RP Console:   Router#monitor session <1-66> <source|destination|filter> <interface|remote|vlan> <FastEthernet|GigabitEthernet|Port-channel|GE-WAN> <tx|rx|both>     SP Console:     Router#remote login switch Router-sp#test monitor session <1-66> <add|del|show> <rp-inband|sp-inband> <tx|rx|both>     -OR-   Router#remote login switch Router-sp#test monitor <add|del|> <session: 1-66> <rp-inband|sp-inband> <tx|rx|both> Router-sp#test monitor session <1-66> <show>   On the 12.2(33)SXH train and later, this is the configuration for an inband sp->rp span session:   Router(config)# monitor session 1 type local Router(config-mon-local)# source cpu  <rp|sp> <tx|rx|both> Router(config-mon-local)# destination interface gigabitethernet 1/2 Router(config-mon-local)# no shutdown   For more information please reference the following link:   http://cco.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.html#wp1109488   Once this information is collected you can then use the source MAC/source IP information to determine the source of the traffic.   Troubleshooting CPU spikes.   At times it is not possible to determine the cause of a CPU spike, since a "show process CPU" cannot be run during the times of the issue.  One way to get around this would be to setup an EEM script to run the command for you when the CPU goes above a certain value.   The following EEM script will run a "show process cpu sorted" when the CPU utilization of the device goes above 50%:   event manager scheduler script thread class default number 1 event manager applet High_CPU event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge entry-val 50 poll-interval 0.5     action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information to <file system>:high_cpu.txt"   action 0.1 cli command "enable"   action 0.2 cli command "show clock | append <file system>:high_cpu.txt"   action 1.2 cli command "term length 0"   action 1.3 cli command "show process cpu sorted | append <file system>:high_cpu.txt"   Please fill in <file system> with the location of the file system without "<" or ">".   If you need further assistance in determining the cause of your high CPU please open a Cisco TAC case. ... View more

    Introduction:   On the Nexus 7000 you may see ICMP packet loss when pinging from the CPU to another device depending on the speed in which this traffic is responded and how much ICMP traffic is being sent to the switch at that moment.  This is due to the default CoPP (Control Plane Policing) service policy that is enabled by default on the N7k.     The following will be seen when attempting to ping another switch from the N7k:   nexus#ping 14.1.106.1 c 1000  PING 14.1.106.1 (14.1.106.1): 56 data bytes 64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=0.852 ms 64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.977 ms 64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=4.101 ms <snip> 64 bytes from 14.1.106.1: icmp_seq=22 ttl=254 time=3.002 ms Request 23 timed out Request 24 timed out Request 25 timed out <snip> --- 14.1.106.1 ping statistics --- 1000 packets transmitted, 987 packets received, 0.98% packet loss round-trip min/avg/max = 0.847/1.112/11.473 ms   The reason for these drops is that ICMP is policed on the N7k very aggressively.  Below is the portion of the CoPP policy which handles ICMP:   IP access list copp-system-acl-icmp          10 permit icmp any any echo          20 permit icmp any any echo-reply   class-map type control-plane match-any copp-system-class-monitoring    match access-group name copp-system-acl-icmp    match access-group name copp-system-acl-icmp6    match access-group name copp-system-acl-traceroute   class copp-system-class-monitoring      set cos 1      police cir 130 kbps bc 1000 ms conform transmit violate drop   We can see that this policy polices ICMP down to 130kbps and that we will drop this traffic if it goes over this amount.   Also keep in mind that these policers are performed a per-module basis.  That means that each module is limited to 130kbps independently.   We can look to see if we are dropping any traffic due to the policy-map with the following command:   7000-1# show policy-map interface control-plane   Below, I am going to pipe for only the monitoring class :   7000-1# sh policy-map interface Control-plane | b monitoring      class-map copp-system-class-monitoring (match-any)        match access-grp name copp-system-acl-icmp        match access-grp name copp-system-acl-icmp6        match access-grp name copp-system-acl-traceroute        set cos 1        police cir 130 kbps , bc 1000 ms        module 2 :          conformed 397890 bytes; action: transmit          violated 12386 bytes; action: drop         module 3 :          conformed 0 bytes; action: transmit          violated 0 bytes; action: drop         module 4 :          conformed 0 bytes; action: transmit          violated 0 bytes; action: drop   As we can see we have violated this policer on module 2 due to this is where 14.1.106.1 is attached:   7000-1# sh ip arp 14.1.106.1   Flags: * - Adjacencies learnt on non-active FHRP router         # - Adjacencies Throttled for Glean         D - Static Adjacencies attached to down interface   IP ARP Table Total number of entries: 1 Address         Age       MAC Address     Interface 14.1.106.1      00:03:45  0000.0c07.acca  Ethernet 2/48   However,  we have not sent any ICMP traffic on any other module, since these are independently policed.   We can remove this order to prove that this is the cause of the drops, we can remove this policy-map for a short period of time and perform the ping again.   However, first lets look at the name of the policy-map used for COPP.  This can be done using the "show run copp" command.    This command will show the entire CoPP policy.  At the bottom of this output will show the same of the service-policy attached to the control-plane:   7000-1# show run copp <snip> control-plane    service-policy input copp-system-policy   Now that we have the name lets remove this policy and attempt the ping again:   7000-1(config)# control-plane 7000-1(config-cp)# no service-policy input copp-system-policy 7000-1(config-cp)#ping 14.1.106.1 c 1000 PING 14.1.106.1 (14.1.106.1): 56 data bytes 64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=1.16 ms 64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.976 ms 64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=0.841 ms <snip>   --- 14.1.106.1 ping statistics --- 1000 packets transmitted, 1000 packets received, 0.00% packet loss round-trip min/avg/max = 0.84/1.131/19.714 ms   As you can see wiwth the CoPP policy removed we are not longer dropping traffic.  At this ping we can put back on the CoPP policy to further protect the CPU.   Also, if we want to leave on the CoPP policy and be able to ping wihtout loss we can adjust the policer to a higher value.  This can be done in the following manner:   7000-1(config)# policy-map type control-plane copp-system-policy  7000-1(config-pmap)# class copp-system-class-monitoring 7000-1(config-pmap-c)#police cir 500 kbps bc 1000 ms conform transmit violate drop   This will increase the amount of ICMP allowed to the CPU to 500kbps per module.  This can be adjusted to your need.   We can see this change with the following command:   F340.11.03-7000-1# sh run copp | b 'control-plane copp-system-policy' policy-map type control-plane copp-system-policy    class copp-system-class-exception      set cos 1      police cir 360 kbps bc 250 ms conform transmit violate drop    class copp-system-class-critical      set cos 7      police cir 39600 kbps bc 250 ms conform transmit violate drop    class copp-system-class-important      set cos 6      police cir 1060 kbps bc 1000 ms conform transmit violate drop    class copp-system-class-management      set cos 2      police cir 10000 kbps bc 250 ms conform transmit violate drop    class copp-system-class-normal      set cos 1      police cir 680 kbps bc 250 ms conform transmit violate drop    class copp-system-class-redirect      set cos 1      police cir 280 kbps bc 250 ms conform transmit violate drop   class copp-system-class-monitoring     set cos 1     police cir 500 kbps bc 1000 ms conform transmit violate drop <snip> control-plane    service-policy input copp-system-policy Related Information: Understand CoPP on Nexus 7000 Series Switches Cisco Nexus 7000 Series Switches Troubleshooting TechNotes Configuring Control Plane Policing ... View more

Troubleshooting high CPU on a 6500 with sup720. The purpose of this document is to cover how to determine the cause of high CPU on a 6500/7600 with a sup720.  The troubleshooting methods discussed in this documentation will make it possible to determine the cause of 90% of all high CPU issues on the sup720. The majority of high CPU on the sup720 is related to CPU usage on the MSFC, thus the majority of this document will cover high CPU on the MSFC. Because it would not be possible to cover every possible reason high CPU can be caused on the sup720, I will demonstrate how to use some of the tools built-in to the sup720 to show general methods on how to narrow down the cause of high CPU. If you are unable to determine the reason based on this documentation, please open a TAC case to investigate this issue further. **Note that these methods can be used to fine high CPU on a RSP720, Sup32 and VS-S720, due to common architecture. Determining Where the CPU utilization is occurring: Within the sup720 6500, there are two types of CPU’s.  One is located on the supervisor and is commonly referred as the SP (Switch Processor) CPU.  The other CPU is located on the MSFC (Multilayer Switch Feature Card) and it commonly referred to as the RP(Route Processor) CPU. Also depending on the module you may also have a DFC (Distributed Feature Card) to perform forwarding locally on that module.  The DFC also has its own CPU, which performs processing locally on the line card.  Under certain scenarios high CPU can be seen on these modules. High CPU on the SP (Switch processor): High CPU on the SP is much less common than high CPU on the RP.   The reasons for high CPU on the SP are typically related to layer 2 operations of the sup720, such a spanning-tree (processing of BPDU's) or processing IGMP snooping/IGMP queries/membership reports as well as LACP/PAGP. You can view the CPU utilization using the following command: SP CPU Util: Switch# remote command switch show process cpu High CPU on the MSFC/RP (Route Processor): This will be traffic that needs to be processed for layer 3 operations, such as ARP, HSRP, forwarding traffic in software.   Below I will go over troubleshooting steps when seeing high CPU on the IP Input/ARP input process as well as CPU utilization caused by interrupt switched traffic on the RP CPU.   You can view the CPU utilization using the following command: RP CPU Util: Switch#show process cpu High CPU on a DFC/module: The CPU on the DFC will help in programming TCAM and router in hardware, since each DFC has its own TCAM. High CPU on a DFC is not very common and can occur for a few different reasons.  One reason you may see high CPU on the DFC is due to Netflow Data Export.   Typically CPU from NDE is expected, but in rare instances it can become high enough to disrupt other processes. You can view the CPU utilization using the following command: DFC CPU Util: Switch# attach <module> Switch-DFC#show process cpu Types of CPU utilization: There are two type of CPU utilization within IoS, interrupt and process. Process based CPU utilization: CPU utilization caused by a process can be caused by few reasons listed below: 1.) Processes switched traffic.  This is traffic that is hitting a specific process in order to be forwarded OR processed by the CPU.  An example of each would traffic being forwarded via the "IP Input" process OR control-plane traffic hitting the "PIM process".  2.) A process trying to clean up tables/previous actions performed.  This can be seen in process such a "CEF Scanner" OR "BGP Scanner", which are used to clean/update the CEF and BGP tables. Interrupt based CPU utilization: CPU caused by an interrupt is always traffic based.  Interrupt switched traffic, is traffic that does not match a specific process, but still needs to be forwarded.   Determining the type of CPU utilization: Process and Interrupt CPU utilization are listed within the "show process cpu" command.   This is broken down below on how to determine what percentage of the CPU utilization is due to interrupt traffic or processed switched traffic: 6500-3#sh proc cpu CPU utilization for five seconds: 0% / 0% ; one minute: 0%; five minutes: 0% Red - Percentage of total CPU utilization Blue - Percentage of the CPU that is caused by Interrupts. Percentage of process CPU util. = Total CPU - Interrupt CPU util. Common reasons for HIGH CPU on the MSFC/RP: IP traffic with a TTL of 1 - Due to the fact we need to send an IP unreachable message to the host letting them know the message has expired in transit.  This cannot be done in hardware and thus the packet must be punted to the MSFC.  Find the device sending traffic the TTL of 1 and stop is from sending traffic, increase the TTL OR install the MLS TTL rate-limiter. Using an ACL with the log keyword - Since a log keyword requires a syslog message to be generated this must be punted to the RP CPU as it cannot be done in hardware. Remove the log keyword from the ACL. Using a PBR route-map without a set statement - Any traffic that matches a PBR route-map with no set statement will be punted.  This is due to the fact that we need to program the next-hop in hardware and if the next-hop is not known, this traffic must be punted to determine the next hop.  Configure a set statement OR remove the policy route from the interface. FIB TCAM Exception -I f you try to install more routes than are possible into the FIB TCAM  you will see the following error message in the logs: CFIB-SP-STBY-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched %CFIB-SP-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched %CFIB-SP-STBY-7-CFIB_EXCEPTION : FIB TCAM exception, Some entries will be software switched This error message is received when the amount of available space in the TCAM is exceeded. This results in high CPU. This is a FIB TCAM limitation. Once TCAM is full, a flag will be set and FIB TCAM exception is received. This stops from adding new routes to the TCAM. Therefore, everything will be software switched. The removal of routes does not help resume hardware switching. Once the TCAM enters the exception state, the system must be reloaded to get out of that state. You can view if you have hit a FIB TCAM exception with the following command: 6500-2#sh mls cef exception status Current IPv4 FIB exception state = TRUE Current IPv6 FIB exception state = FALSE Current MPLS FIB exception state = FALSE When the exception state is TRUE, the FIB TCAM has hit an exception. The maximum routes that can be installed in TCAM is increased by the mls cef maximum-routes command. This issue is common when trying to route a full BGP table on PFC-3A or a PFC-3B. **Note a failover of the supervisors in dual supervisor system will not recover this exception, even through the “show mls cef exception status” will no longer indicate a FIB exception.  A full reload of the switch is required. ICMP redirects - If traffic is taking a path that is not efficient, an ICMP redirect will be sent out to inform the host of a better next-hop.  This will cause the packet to be punted in order to trigger the MSFC to send the ICMP redirect to the host.   On the 6500 ICMP redirects will be sent to the CPU with a destination index of 0x7F07.  This can be seen when performing a netdr capture. Turn off icmp redirects to stop this traffic from being punted.  However this is an indication of network inefficiency that was attempting to be dynamically resolved.   User interaction is needed in order to track down this inefficiency.  If you need assistance in determining why ICMP redirects are being generated please open a TAC case. CEF Glean adjacency   - This can happen when no ARP resolution for the next hop.  All traffic must be punted in order to trigger an ARP request for the next hop.  This will always manifest it self as interrupt based traffic. To protect the RP CPU from this issue you can implement the Glean adj. mls rate-limiter. Netflow and ACL feature configured on the same interface matching the same traffic - You cannot have an ACL based feature and a Flow based feature configured on the same interface for the same traffic.  An Example of this would be having NAT and PBR configured on the same interface matching the same traffic. NAT is netflow assisted, as the first packet in every flow would need to be punted to create the netflow entry in hardware.  Once the netflow entry is created all subsequent packets will hit this hardware netflow entry and thus be forwarded in hardware. Policy-Based Routing is ACL based. This will create a “policy-route” state when a route-map is configured to use PBR and applied to that interface.   This will point to a special adjacency, which is where the next-hop is specified in the “set” statement of the route-map. The issue comes when a packet matches both the NAT and the PBR feature, the traffic can not be sent to the CPU to be put into Netflow AND be redirected to the PBR special adj, thus this traffic must be software switched.  If these two features overlap, these features are taken out of hardware and the traffic is software switched.  When this occurs neither feature may be applied to the matching traffic.  If a packet does not match both the ACL based feature and Netflow based feature match criteria then the relevant function (ACL based or Netflow based) will be performed in hardware. Therefore, for proper hardware based performance in situations where ACL based features and Netflow based features are configured on the same interfaces it is important to have unique policies. In order to look at this information in feature manager and ACL TCAM please refer to section “ Determining features programmed on a layer 3 interface ” below. To work around this problem do not have both an ACL based and Netflow based feature configured on the same interface, matching the same traffic. **Note- For a more indepth look at how features are programmed on a interface please go to the Determining the reason why traffic is punted to the CPU, section. Directed Broadcast traffic – All broadcast traffic must be sent to the MSFC on a vlan when a layer 3 interface is configured within that vlan.  This includes directed broadcast traffic.   Use multicast instead of directed broadcast. Bridging loop - If a bridging loop occurs on the network, this could cause high CPU on the MSFC. All broadcast traffic must be sent to the MSFC on a vlan when a layer 3 interface is configured within that vlan. You can determine what traffic is hitting the CPU by using a netdr capture to track down the source interface of the loop (See Using Netdr to determine traffic punted to the CPU section).  Determining the source of the CPU utilization: Determine the source of RP CPU utilization using interface buffers: **Note** you will only be able to see traffic in the interface buffers on a layer 3 interface if the traffic is being processed switched (see “Determining type of CPU utilization” above).  This will not work when traffic is being interrupt switched.  In the case of interrupt switched traffic use the netdr capture instead. One of the quickest ways to determine the layer 3 interface that is the source of traffic that is causing high CPU is to see which interface has a large amount of drops flushes on the interfaces input queue.  The input queue on a layer 3 interface is the CPU queue for that interface on the sup720.   If we ever see packets/drops on the input queue on the sup720 it is always due to traffic that is being sent towards the CPU.  You can narrow down the location of such an interface with the following commands: 6500-2#show interface | include  is up|drop Vlan10 is up, line protocol is up   Input queue: 74/75/18063/18063 (size/max/drops/flushes); Total output drops: 0 Vlan20 is up, line protocol is up Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 We can see that SVI (Switched Virtual Interface) 10 has 74 packets in its buffer, whose queue size is 75 packets. This demonstrates that a large amount of traffic is being punted on this interface to the RP CPU, since this queue is full. Now that we can see a large amount of traffic within this queue, we can look at what is in this queue with the command "show buffers input-interface vlan 10 header".  This command will display the IP header of the packet so we can attempt to determine the source.  If you want to look at the entire packet you can use the command  "show buffers input-interface vlan 10 packet". Below is the output from this command for SVI 10 6500-2#sh buffers input-interface vlan 10 header Buffer information for Small buffer at 0x4667A08C   data_area 0x802F664, refcount 1, next 0x466AE968, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x802F6DA, datagramsize 60, maximum size 308   mac_start 0x802F6DA, addr_start 0x802F6DA, info_start 0x0   network_start 0x802F6E8, transport_start 0x802F6FC, caller_pc 0x41F78790   source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,   TOS: 0 prot: 6, source port 0, destination port 0 Above we can see the basic information about this traffic that is included in the IP header, including the TOS, TTL and protocol encapsulated within the IP header. If we viewed the entire packet we can look at more in depth information including the layer 2 information, as can be seen below: 6500-2#sh buffers input-interface vlan 10 packet Buffer information for Small buffer at 0x466A23B0   data_area 0x80340A4, refcount 1, next 0x466E991C, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x52836BE4 (Vlan10), if_output 0x0 (None)   inputtime 16:32:10.292 (elapsed 00:00:50.608)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x803411A, datagramsize 60, maximum size 308   mac_start 0x803411A, addr_start 0x803411A, info_start 0x0   network_start 0x8034128, transport_start 0x0, caller_pc 0x41F78790   source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,   TOS: 0 prot: 6, source port 0, destination port 0        0: 0015C726 FB80 0000 01000600 0800 4500  ..G&{.........E.       16: 002E0000 00000106 3651 0A0A 0A02 0A64   ........6Q.....d       32: 650A 0000 00000000 00000000 00005000  e.............P.       48: 0000265C 00000001 02030405 FD        ..&\........}             Red = Source MAC Blue = Dest MAC Green = Ethertype (0x800 for IP traffic) Purple = Src. IP Orange = Dest IP Using Netdr to determine traffic punted to the CPU: A netdr capture is preformed on the MSFC CPU controller.  This is the closest location you can capture a packet on the MSFC in order to determine why traffic is being punted to the MSFC/RP CPU.  With a Sup720 or Sup32 it allows one to capture packets on the RP or SP inband. The netdr command can be used to capture both Tx and Rx packets in the software-switching path. Cat6500#debug netdr capture ? acl                     (11) Capture packets matching an acl and-filter              (3) Apply filters in an and function: all must match continuous              (1) Capture packets continuously: cyclic overwrite destination-ip-address  (10) Capture all packets matching ip dst address dstindex                (7) Capture all packets matching destination index ethertype               (8) Capture all packets matching ethertype interface               (4) Capture packets related to this interface or-filter               (3) Apply filters in an or function: only one must match rx                      (2) Capture incoming packets only source-ip-address       (9) Capture all packets matching ip src address srcindex                (6) Capture all packets matching source index tx                      (2) Capture outgoing packets only vlan                    (5) Capture packets matching this vlan number <cr> OPTIONS: Using the continuous option, the switch will capture packets on the RP-inband continuously fill the entire capture buffer (4096 packets) and then start to overwrite the buffer in a FIFO fashion. The tx and rx options will capture packets coming from the MSFC and going to the MSFC respectively. The and-filter and the or-filter specify that an and or an or will be applied respectively to all of the options that follow. For example, if you use the syntax below, then both option #1 and option #2 must match for the packet to be captured. Similarly, if the or-filter is used either option #1 or option #2 or both must match for the packet to be captuered. debug netdr and-filter option#1 option#2  The interface option is used to capture packets to or from the specified interface. The interface can be either an SVI or a L3 interface on the switch. The vlan option is used to capture all packets in the specified VLAN. The VLAN specified can also be one of the internal VLANs associated with a L3 interface. The srcindex and dstindex options are used to capture all packets matching the source ltl and destination ltl indices respectively. Note that the interface option above only allows the capture of packets to or from a L3 interface (SVI or physical). Using the srcindex or dstindex options allows the capture of Tx or Rx packets on a given L2 interface. The srcindex and dstindex options work with either L2 or L3 interface indices. The ethertype option allows the capture of all packets matching the specified ethertype. The source-ip-address and destination-ip-address options allow the capture of all packets matching the specified source or destination IP address respectively. Below is an example of capturing traffic destined to 10.100.101.10 sourced from 10.10.10.2 going to the RP CPU: 6500-2#debug netdr cap rx and-filter source-ip-address 10.10.10.2 destination-ip-address 10.100.101.10 6500-2#sh netdr cap A total of 4096 packets have been captured The capture buffer wrapped 0 times Total capture capacity: 4096 packets ------- dump of incoming inband packet ------- interface Vl10 , routine mistral_process_rx_packet_inlin, timestamp 00:00:11 dbus info : src_vlan 0xA(10), src_indx 0xC0 (192), len 0x40(64)   bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)   10020400 000A0000 00C00000 40080000 00060468 0E000040 00000000 03800000 mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)   requeue 0, obl_pkt 0, vlan 0xA(10) destmac 00.15.C7.26.FB.80, srcmac 00.00.01.00.06.00, protocol 0800 protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 0   df 0, mf 0, fo 0, ttl 100, src 10.10.10.2, dst 10.100.101.10      tcp src 0, dst 0, seq 0, ack 0, win 0 off 5 checksum 0x265C Red = Ingress Vlan of traffic Blue = Layer 3 interface traffic is coming from Green = Ethertype and SRC/DST MAC addresses Purple = IP Header Orange = SRC index (source of ingress traffic). Dark Red = Dest Index (where traffic is being sent). Using the above information you should be able to track down the host or interface that You can track down the physical port where this traffic is sourced from by tracking down the mac-address (which is the last device to own this traffic).   This can be done in the following manner: -6500-2#show mac-address address 0000.0100.0600 Legend: * - primary entry         age - seconds since last seen         n/a - not available   vlan   mac address     type    learn     age              ports ------+----------------+--------+-----+----------+-------------------------- Active Supervisor: *   10  0000.0100.0600   dynamic  Yes          0   Gi4/1 You can look at ingress location of this traffic by taking the source index and seeing what interface it references: switch#remote command switch test mcast ltl index c0 index 0xC0 contain ports 4/1 We can see this traffic is coming from port 4/1, which is confirmed based on the port where the MAC is learned.  This can be very useful when determining the source of the traffic during bridging loop, if multiple types of traffic is being flooded, but sent from the same port. The same can also be done to determine where this traffic is being sent.  In this case we can see that the traffic is being sent to the destination index of 0x380, which is the RP CPU: 6500-2#rem com sw test mcast ltl ind 380 index 0x380 contain ports 2/R 2/R = module/Router If the destination has the flood bit set in the netdr capture, this traffic is being flooded to all ports within the vlan.   You will need to change the destination index to account for this bit setting.  In order to see what ports this traffic Is flooded you will need to add 8 to the beginning of the index.   Example: When the flood bit is equal to 1: If the index was 0x400A this could change to 0xC00A. An index of 0xC12 would become 0x8C12. With the two cases above the Flooding would be occurring in vlan 10 (A = 10) and vlan 3090 (c12= 3090). Once we find the source of the traffic, we can see why this traffic is being punted.  The punt to the RP CPU will always occur via a layer 3 interface. Using a CPU SPAN to determine traffic being punted to the CPU This capture is performed on the ASIC, which is connected to the RP/SP CPU.    This will allow you to replicate traffic that is being sent to the RP or SP CPU to a capture device.   This can be handy for determine the cause of the HIGH CPU OR determining if traffic is being sent to or from the CPU for processing (such as HSRP/OSPF/PIM control plane traffic).   When using the 12.2(18)SXF train and earlier the  configuration for an inband span session is as follows: RP Console: Router#monitor session <1-66> <source|destination|filter> <interface|remote|vlan> <FastEthernet|GigabitEthernet|Port-channel|GE-WAN> <tx|rx|both> SP Console: Router#remote login switch Router-sp#test monitor session <1-66> <add|del|show> <rp-inband|sp-inband> <tx|rx|both> -OR- Router#remote login switch Router-sp#test monitor <add|del|> <session: 1-66> <rp-inband|sp-inband> <tx|rx|both> Router-sp#test monitor session <1-66> <show> On the 12.2(33)SXH train and later, this is the configuration for an inband sp->rp span session: Router(config)# monitor session 1 type local Router(config-mon-local)# source cpu  <rp|sp> <tx|rx|both> Router(config-mon-local)# destination interface gigabitethernet 1/2 Router(config-mon-local)# no shutdown For more information please reference the following link: http://cco.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/span.html#wp1109488 Once this information is collected you can then use the source MAC/source IP information to determine the source of the traffic. Troubleshooting CPU spikes. At times it is not possible to determine the cause of a CPU spike, since a "show process CPU" cannot be run during the times of the issue.  One way to get around this would be to setup an EEM script to run the command for you when the CPU goes above a certain value.   The following EEM script will run a "show process cpu sorted" when the CPU utilization of the device goes above 50%: event manager scheduler script thread class default number 1 event manager applet High_CPU event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge entry-val 50 poll-interval 0.5   action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information to <file system>:high_cpu.txt"   action 0.1 cli command "enable"   action 0.2 cli command "show clock | append <file system>:high_cpu.txt"   action 1.2 cli command "term length 0"   action 1.3 cli command "show process cpu sorted | append <file system>:high_cpu.txt" Please fill in <file system> with the location of the file system without "<" or ">". Determining the reason why traffic is punted to the CPU. Once we determine what traffic is punted to the CPU, determining the why requires a little more digging.  You can use the following commands to help determine why traffic is being punted. Looking at the ACL TCAM on a Layer 3 interface: Under normal scenarios we should not see any punt adjacencies installed in ACL TCAM on the interface.  This can be determined by looking at the ACL TCAM for the layer 3 interface. Example: 6500-2#show run interface vlan 10 Building configuration... Current configuration : 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 end Lets take a look at the ACL TCAM in this configuration 6500-2#show tcam interface vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     permit       ip any any We can see that all traffic is being forward in hardware due to the "permit ip any any".  We can see that since we have no features programmed on egress that nothing will be seen in ACL TCAM: 6500-2#show tcam interface vlan 10 acl out ip Now if we applied the following access-list with a log keyword on input we will see the following: 6500-2#sh access-list 1 Standard IP access list 1     10 permit 10.10.10.20 log     20 permit any 6500-2#sh run int vlan 10 Building configuration... Current configuration : 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip access-group 1 in end 6500-2#sh tcam interface vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     punt         ip host 10.10.10.20 any     permit       ip any any Now we can see a punt installed for any traffic from host 10.10.10.20, due to the log keyword on input.  The punt represents traffic that needs to be sent to the RP CPU.    In this case, all this traffic that is sources from 10.10.10.20 will be sent to the RP CPU to be processed switched. Keep in mind that this is only configured on ingress.  If we look at the egress ACL TCAM programming, we can see that this has not been adjusted. 6500-2#show tcam interface vlan 10 acl out ip Determining features programmed on a layer 3 interface: Lets take the same interface and see what FM (Feature Manager) says about the features configured on this interface: 6500-2#sh access-list 1 Standard IP access list 1     10 permit 10.10.10.20 log     20 permit any 6500-2#sh run int vlan 10 Building configuration... Current configuration : 85 bytes ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip access-group 1 in end Lets take a look at FM with a ACL configured. 6500-2#sh fm fie interface vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : RACL   - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = RACL  +-------------------------------------+             Action Merge Table +-------------------------------------+    RACL               RSLT              R_RSLT            COL +-------------------------------------+    L2R              L2R               P                 0               SB               HB                P                 0               HB               HB                P                 0               L3D              L3D               L3D               0               P                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 2   num# of free TCAM entries in Bank2 = 32758 Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB           HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32757 Interface Vl10 [Egress]: No Features Configured No IP Guardian Feature Configured No IPv6 Guardian Feature Configured No QoS Feature Configured Above we can see that the feature “RACL” (Routed ACL) is configured as ingress feature. We can see that this has invoked FM label 52 to be applied to this interface.  FM decides this label based on the features configured as well the flowmask configured. We can also see the IP Guardian has not been triggered.  IP Guardian determines what flowmask is needed on an interface for the features configured.  This is used when netflow assistance is needed for a feature.  It will also protect from having incompatible features programmed into hardware. Some features require specific flowmasks in order to be put into hardware. This can be seen at the following link: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/netflow.html#wp1132875 If no flowmask is configured, the 6500 will dynamically choose the appropriate flowmask via IP guardian. On Egress we can see that no features have been configured. Feature Conflict: If we had a feature conflict, such as if we had PBR and NAT configured on the same interface it would look like the following: ip nat pool test 10.10.101.2 10.10.101.20 prefix-length 24 ip nat inside source list 7 pool test access-list 7 permit 10.10.10.0 0.0.0.255 route-map PBR permit 10 match ip address 1 set ip next-hop 10.10.101.1 ! access-list 1 permit 10.10.10.0 0.0.0.255 ! interface Vlan10 ip address 10.100.101.1 255.255.255.0 ip nat inside ip policy route-map PBR end ! interface Vlan20 ip address 10.10.101.2 255.255.255.0 ip nat outside end Looking at FM to see what these incompatible features would look like: 6500-2#show fm fie interface vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : PBR   - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank1 = PBR  +-------------------------------------+             Action Merge Table +-------------------------------------+    PBR                RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               AdR              AdR               P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 12   num# of free TCAM entries in Bank1 = Unknown   num# of merged VMRs in bank 2 = 0   num# of free TCAM entries in Bank2 = Unknown Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32760 Interface Vl10 [Egress]: No Features Configured IP Guardian Feature Configured Current IP Guardian flowmask : Intf Full Flow FIE guardian interaction : Done FIE guardian interaction result : SUCCESS FIE guardian flowmask conflict status : FIE_FLOWMASK_NO_CONFLICT num# of features that bump this interface : 1 features that bump this interface : NAT No IPv6 Guardian Feature Configured No QoS Feature Configured We can see the IP guardian has selected to use the “Interface Full Flow” flowmask due to the NAT feature being configured on this interface. We can also see that NAT is causing this interface to “bump”.  This means that this feature is needs to be software switched, or netflow assisted.  This could cause traffic to be punted OR a netflow entry to created in hardware that will be used. ***Note  netflow always takes precedence to whatever is configured in ACL TCAM.  Typically a “redirect” in hardware  will demonstrate this traffic is being pushed to the netflow path.  However, this could also be seen a punt.  Please check for a netflow entry for the traffic if a “punt “ is installed and the feature is a netflow assisted feature. We can also take a look at what the TCAM would look like with these features configured. 6500-2#sh tcam int vlan 10 acl in ip * Global Defaults shared Entries from Bank 0 Entries from Bank 1     redirect     ip any any fragments     redirect     ip any any fragments     redirect     ip any any     permit       ip any 224.0.0.0 15.255.255.255 fragments     permit       ip any 224.0.0.0 15.255.255.255 fragments     permit       ip any 224.0.0.0 15.255.255.255     policy-route ip 10.10.10.0 0.0.0.255 any fragments     policy-route ip 10.10.10.0 0.0.0.255 any fragments     policy-route ip 10.10.10.0 0.0.0.255 any     permit       ip any any fragments     permit       ip any any fragments     permit       ip any any We can see that a redirect is installed for NAT traffic.   We also see the “policy-route” statement installed, which is trying to redirect the same traffic that is attempting to have NAT applied.  The redirect installed for NAT will cause this traffic to be redirected to the CPU for a netflow entry to be installed.  However, If we look at netflow we will see the netflow entry installed you will see that no traffic is hitting the entry that was created in hardware: 6500-2#sh mls netflow ip Displaying Netflow entries in Active Supervisor EARL in module 2 DstIP           SrcIP           Prot:SrcPort:DstPort  Src i/f          :AdjPtr ----------------------------------------------------------------------------- Pkts         Bytes         Age   LastSeen  Attributes --------------------------------------------------- 0.0.0.0         0.0.0.0         0   : 0    : 0        --               :0x0         30           1380          72    13:31:25   L3 - Dynamic 10.10.102.10    10.10.10.2      tcp : 0     : 0   Vl10             :0x0         0            0             82    13:31:25   L3 - Dynamic We can see in red above that not packets/bytes have hit this entry.  This is because this traffic is being software forwarded.  This is due to the fact that PBR is configured for the same subnet (10.10.10.x).   We can also see that the CPU is high: 6500-2#sh processes cpu CPU utilization for five seconds: 57%/57%; one minute: 13%; five minutes: 7% **Note all this will cause the CPU to be drive by interrupts not a process. As soon as we remove PBR we can see that this entry is not being hit: 6500-2(config)#int vlan 10 6500-2(config-if)#no ip policy route-map PBR 6500-2(config-if)#do sh mls netflow ip Displaying Netflow entries in Active Supervisor EARL in module 2 DstIP           SrcIP           Prot:SrcPort:DstPort  Src i/f          :AdjPtr ----------------------------------------------------------------------------- Pkts         Bytes         Age   LastSeen  Attributes --------------------------------------------------- 10.10.102.10    10.10.10.2      tcp : 0     : 0    Vl10             :0x80002     77239        3552994       6     13:36:03   L3 - SwInstalled 0.0.0.0         0.0.0.0         0   : 0     : 0      --               :0x0         126          5796          351   13:35:57   L3 - Dynamic We can also see that the CPU drops: 6500-2#sh proc cpu | ex 0.00 CPU utilization for five seconds: 0%/0%; one minute: 45%; five minutes: 20% Though FM still shows that NAT is bumping this interface we can now see that the hardware installed netflow entry is being hit.  We can also see that PBR is not longer listed as an installed feature and is now listed as “empty”: 6500-2#sh fm fie int vlan 10 Interface Vl10: Feature interaction state created: Yes Flowmask conflict status for protocol IP : FIE_FLOWMASK_STATUS_SUCCESS Flowmask conflict status for protocol OTHER : FIE_FLOWMASK_STATUS_SUCCESS Interface Vl10 [Ingress]: Slot(s) using the protocol IP : 2 FIE Result for protocol IP : FIE_SUCCESS_NO_CONFLICT Features Configured : [empty] - Protocol : IP FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 num# of strategies tried : 1   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = Unknown   num# of merged VMRs in bank 2 = 6   num# of free TCAM entries in Bank2 = Unknown Slot(s) using the protocol OTHER : 2 FIE Result for protocol OTHER : FIE_SUCCESS_NO_CONFLICT Features Configured : OTH_DEF   - Protocol : OTHER FM Label when FIE was invoked : 52 Current FM Label : 52 Last Merge is for slot: 0 Features in Bank2 = OTH_DEF  +-------------------------------------+             Action Merge Table +-------------------------------------+    OTH_DEF            RSLT              R_RSLT            COL +-------------------------------------+    SB               HB                P                 0               X                P                 P                 0            +-------------------------------------+ num# of strategies tried : 1 Description of merging strategy used:   Serialized Banks: FALSE   Bank1 Only Features: [empty]   Bank2 Only Features: [empty]   Banks Swappable: TRUE Merge Algorithm: ODM   num# of merged VMRs in bank 1 = 0   num# of free TCAM entries in Bank1 = 32730   num# of merged VMRs in bank 2 = 1   num# of free TCAM entries in Bank2 = 32746 Interface Vl10 [Egress]: No Features Configured IP Guardian Feature Configured Current IP Guardian flowmask : Intf Full Flow FIE guardian interaction : Done FIE guardian interaction result : SUCCESS FIE guardian flowmask conflict status : FIE_FLOWMASK_NO_CONFLICT num# of features that bump this interface : 1 features that bump this interface : NAT No IPv6 Guardian Feature Configured No QoS Feature Configured Using “show ip traffic” statistics to see why traffic is punted: 6500-2#show interface | i is up|drop   Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Vlan10 is up, line protocol is up   Input queue: 74/75/18063/18063 (size/max/drops/flushes); Total output drops: 0 we can see a large amount of traffic in the vlan 10 queue. Vlan20 is up, line protocol is up SVI 10 is receiving a large amount of traffic that is being punted to the RP CPU.  When we look at what is in this queue with the command "show buffers input-interface vlan 10 header". Below is the output from this command for SVI 10 6500-2#sh buffers input-interface vlan 10 header Buffer information for Small buffer at 0x4667A08C   data_area 0x802F664, refcount 1, next 0x466AE968, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x802F6DA, datagramsize 60, maximum size 308   mac_start 0x802F6DA, addr_start 0x802F6DA, info_start 0x0   network_start 0x802F6E8, transport_start 0x802F6FC, caller_pc 0x41F78790   source: 10.10.10.2, destination: 10.100.101.10, id: 0x0000, ttl: 1,   TOS: 0 prot: 6, source port 0, destination port 0 Buffer information for Small buffer at 0x4667C7E8   data_area 0x80314A4, refcount 1, next 0x46695FD0, flags 0x200   linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1   if_input 0x530D5048 (Vlan10), if_output 0x0 (None)   inputtime 00:00:00.000 (elapsed never)   outputtime 00:00:00.000 (elapsed never), oqnumber 65535   datagramstart 0x803151A, datagramsize 60, maximum size 308   mac_start 0x803151A, addr_start 0x803151A, info_start 0x0   network_start 0x8031528, transport_start 0x803153C, caller_pc 0x41F78790   source: 10.10.10.1, destination: 10.10.10.2, id: 0xD096, ttl: 255, prot: 1 Since at this point we are unsure why this traffic is being punted, we can look at “show ip traffic" statistics to see why this traffic is being punted to the CPU.  First start by clearing the IP traffic statistics.  We can then see what is incrementing in these counters to see what would be the cause: 6500-2#clear ip traffic Clear "show ip traffic" counters [confirm] 6500-2#sh ip traffic IP statistics:   Rcvd:  33516 total, 0 local destination          0 format errors, 0 checksum errors, 33516 bad hop count <------We can see that the bad Hop count in this case is incrementing          0 unknown protocol, 0 not a gateway          0 security failures, 0 bad options, 0 with options   Opts:  0 end, 0 nop, 0 basic security, 0 loose source route          0 timestamp, 0 extended security, 0 record route          0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump          0 other   Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble          0 fragmented, 0 couldn't fragment   Bcast: 0 received, 0 sent   Mcast: 0 received, 0 sent   Sent:  0 generated, 0 forwarded   Drop:  40005 encapsulation failed, 0 unresolved, 0 no adjacency          0 no route, 0 unicast RPF, 0 forced drop          0 options denied, 0 source IP address zero ICMP statistics:   Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable         0 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench         0 parameter, 0 timestamp, 0 info request, 0 other         0 irdp solicitations, 0 irdp advertisements         0 time exceeded, 0 timestamp replies, 0 info replies   Sent: 0 redirects, 0 unreachable, 0 echo, 0 echo reply         0 mask requests, 0 mask replies, 0 quench, 0 timestamp         0 info reply, 58464 time exceeded, 0 parameter problem <---- We can also the 6500 is sending ICMP TTL expired messages as well.         0 irdp solicitations, 0 irdp advertisements <snip> Looking at the traffic statistics we can see that the bad hop count counter is incrementing and the switch is sending ICMP time exceeded messages.  On the 6500 all traffic with a TTL of 1 is punted to the CPU so that an ICMP TTL expired message can be sent to the host who sent this traffic.  The first packet in the buffer can be seen to have TTL of 1, which is why this traffic is punted.  We can see that the 2nd packet is sourced from 10.10.10.1 (SVI 10) sent to 10.10.10.2. This packet is an ICMP TTL expired message. Looking at the InBand Channel between the SP and the RP.  The IBC on the sup720 is used for traffic that needs to go to the MSFC for processing. Below we can see the rate of traffic being sent to the RP CPU.  This should be a relatively low value as it should only be control plane traffic such a PIM, OSPF, HRSP, etc.  If we see a high rate of traffic here, then we know we are punting traffic towards the RP CPU on the SP -> RP inband channel. If we see a large amount of traffic being received on the IBC (towards the MSFC from the SP) this could be an indication that traffic is being punted. F340.05.04-6500-2# sh ibc Interface information:             Interface IBC0/0(idb 0x5153A9B8)             Hardware is Mistral IBC (revision 5)             0 minute rx rate 0 bits/sec, 1 packets/sec             0 minute tx rate 1000 bits/sec, 1 packets/sec             233965095 packets input, 14038036500 bytes             0 broadcasts received             27419 packets output, 2176622 bytes             14 broadcasts sent             0 Inband input packet drops             0 Bridge Packet loopback drops             6893268 Packets out of 6893268 CEF Switched, 0 Packets out of 0 Tag CEF Switched             0 Packets Fast Switched             0 Packets SLB Switched, 0 Packets CWAN Switched             Potential/Actual paks copied to process level 227045667/50587903 (176457764 dropped, 0 spd drops)             1259 paks copied at vlan soutput             0 paks copied at ibc fast soutput             0 paks copied at ibc fastsend             22773678 inband interrupts             39 transmit ring cleanups             25805398 ibl inputs         39 total tx interrupts set                 39 tx ints due to packets outstanding                 0 tx ints due low free buffers in pool                 0 tx ints due to application setting             tx dma done batch size=32             buffers free minimum before tx int=4             mistral ran out of tx descriptors 0 times             mistral tx interrupt inconsisteny occured 0 times             Label switched pkts dropped: 0             Xconnect pkts processed: 0, dropped: 0             IBC resets   = 2; last at 01:37:42.747 UTC Tue Mar 1 2011 <snip> You can also look at this command to look for any drops of traffic that is forwarded towards the CPU.   These counters are highlighted above in blue. ... View more

    Introduction:   DHCP Relay Agent: Routers, by default, will not forward broadcast packets. Since DHCP client messages use the destination IP address of 255.255.255.255 (all Nets Broadcast), DHCP clients will not be able to send requests to a DHCP server on a different subnet unless the DHCP/BootP Relay Agent is configured on the router. The DHCP/BootP Relay Agent will forward DHCP requests on behalf of a DHCP client to the DHCP server. The DHCP/BootP Relay Agent will append its own IP address to the source IP address of the DHCP frames going to the DHCP server. This allows the DHCP server to respond via unicast to the DHCP/BootP Relay Agent. The DHCP/BootP Relay Agent will also populate the Gateway IP address field with the IP address of the interface on which the DHCP message is received from the client. The DHCP server uses the Gateway ip address field to determine the subnet from which the DHCPDISCOVER, DHCPREQUEST, or DHCPINFORM message originates.   NX-OS's implementation of a DHCP relay on a Nexus 7000 differs from that of how we implement the DHCP helper address on the 6500 and other IoS based platforms.   On the Nexus 7000 we use an VACL to redirect all DHCP broadcast traffic to the CPU when the DHCP relay function is implemented.  When this redirect occurs the Nexus 7000 does NOT broadcast these DHCP broadcast packets to all ports within the VLAN as one would expect.  This is done based on the fact that it is believed that if a DHCP relay agent is configured, than the DHCP server does not exist on the local vlan and there is no need to broadcast this traffic locally.   This is different than what occurs on the 6500.  On the 6500 all DHCP broadcast are leaked to the CPU and flooded to all ports within the vlan.   On the N7k you can view the redirect VACL in hardware via the following command, 'show system internal access-list vlan X input statistics'.  Below shows what the ACL redirect  would look like for a relay agent configrued on SVI 10.  This only shows the redirect VACL for module 2.  If you have mulitple modules in the device, this VACL redirect would be configured for every module (this is because there is a forwarding engine on every module on a N7k):   7000-1# sh system internal access-list vlan 10 input statistics   slot  2 =======   Tcam 1 resource usage: ---------------------- Label_b = 0x801 Bank 0 ------     IPv4 Class       Policies: DHCP Snooping()  [Merged]       Entries:         [Index] Entry [Stats]         --------------------- [0003] redirect(0x800) udp 0.0.0.0/0 255.255.255.255/32 eq 68  [0] [0004] redirect(0x800) udp 0.0.0.0/0 255.255.255.255/32 eq 67  [0] [0005] redirect(0x800) udp 0.0.0.0/0 eq 68 255.255.255.255/32  [0] [0006] redirect(0x800) udp 0.0.0.0/0 eq 67 255.255.255.255/32  [0] [0007] permit ip 0.0.0.0/0 0.0.0.0/0  [0]   The issue with not broadcasting the DHCP discover when a relay agent is configured is that in certain scenarios this can break functionality, when a local servers need to hear the DHCP broadcast packet.  This can also occur if the server is not directly attached to the N7k, but to get to the server the traffic must pass through a N7k with the DHCP relay agent configured for that vlan. In order to get around scenarios where the local DHCP discover needs to be heard by a specific device on the local subnet as well as the DHCP server an additional DHCP relay-agent address would need to be configured on the SVI of that vlan.  This relay-agent address would need to point to the local device that needs to hear this traffic.  Since DHCP traffic is redirected to all relay-agents configured the DHCP discover would be sent to the DHCP server(12.3.4.6) as well as the local server(10.0.0.2).  An example of this can be been below:   Interface vlan 12 ip address 10.0.0.1 255.255.255.0 ip dhcp relay 12.3.4.6  <---DHCP server ip dhcp relay 10.0.0.2 <---IP address of local server.   If multiple devices on that subnet need to hear this traffic, you can specify a local a relay-agent as the directed broadcast address of the local subnet.  This will cause the DHCP traffic to be flooded throughout the local subnet (as destination 10.0.0.255) as well as being sent to the DHCP server(12.3.4.6):   Interface vlan 12 ip address 10.0.0.1 255.255.255.0 ip dhcp relay 12.3.4.6  <---DHCP server ip dhcp relay 10.0.0.255 <---Local subnet broadcast   Since the relay-agent is simply forwarding this traffic to the configured address, this will allow this traffic to get to the correct location.   Below is a step-by-step walk through of the DHCP Process on the N7k and 6500 discussing the difference above.   DHCP relay-agent process on the N7k:   DHCP discover is sent by the client When this packet ingresses the N7k it is redirected to the CPU/relay agent by the ingress forwarding engine and is not broadcast on the vlan. The relay agent redirects the DHCP discover to the DHCP server. DHCP server responds to the IP address of the relay-agent in the  giaddr field of the discover/request packet with a DHCP offer, which is relayed to the client. DHCP client responds with DHCP request (whether it be an unicast or broadcast response), this is redirected to the relay-agent and forwarded on to the DHCP server. DHCP server ACK's the request, which is sent to the relay-agent.  This is then relayed to the client, finishing the DHCP process.   DHCP helper address process on the 6500:   DHCP discover is sent by the client When this packet ingresses the 6500 it is leaked to the CPU/relay agent based on a destination index, which includes the vlan broadcast. The relay agent redirects the DHCP discover to the DHCP server. DHCP server responds to the IP address of the relay-agent in the giaddr field of the discover/request packet with a DHCP offer, which is relayed to the client. DHCP client responds with DHCP request (whether it be an unicast or broadcast response), this is redirected to the relay-agent and forwarded on to the DHCP server. DHCP server ACK's the request, which is sent to the relay-agent.  This is then relayed to the client, finishing the DHCP process. ... View more