Souvik Ghosh is a CSE at the Cisco TAC in Bangalore, India. He has 3.5 years of experience in LAN switching technologies. LAN switching products such as the Cisco Catalyst 6k,4k,2k Series Switches are his areas of expertise. He has been involved in various escalation requests from India, Singapore & Australia & is currently working as a tech lead for the LAN switching team in Bangalore, India. He holds CCNP and CCIP certifications.   Download the slides in PDF format here, read the FAQ here ... View more

  Introduction   Souvik Ghosh is a customer support engineer at the Cisco Technical Assistance Center in Bangalore, India. He has three and half years of experience in LAN switching technologies. LAN switching products such as the Cisco Catalyst 6500, 4500, 3750, and 2960 Series Switches are his areas of expertise. He has been involved in various escalation requests from India, Singapore, and Australia and is currently working as a technical lead for the LAN switching team in Bangalore, India. He holds CCNP and CCIP certifications.     The following experts were helping Souvik to answer few of the questions asked during the session: Amit Singh , Akshay Balaganur , and Ranganatha Raju . Amit, Akshay and Ranganatha are support engineers and have vast knowledge in 6500 related topics.     You can download the slides of the presentation in PDF format here. The related Ask The Expert sessions is available here. The Complete Recording of this live Webcast can be accessed here.   General Questions Q. We have an issue with the Catalyst 6500 in VSS and broadcast coming from a checkpoint cluster. As a result, there is high CPU on Catalyst 6500. Is there any way to drop broadcast traffic and not make it go to the control plane? A. This depends totally on the kind of broadcast. We can implement storm control on access layer switches so that we can save the CPU in case of broadcast storm. In case of ARP there are two options available: one is implementing mls qos protocol arp command, which will rate limit arp packets in hardware. Second, we can implement ip dhcp snooping option along with arp inspection and configure rate limit to arp inspection value. The drawback to both of these is that they are system wide, so normal broadcast packets are also dropped. As a result, storm control is the best option.   Q. Is it normal to see CPU usage around 60%?­ A. 60% constant CPU usage is not normal. However, an intermittent spike might occur and be normal. It also depends on if you are using Non-modular IOS (mz image) or Modular IOS (vz image).   Q. I have a problem with a sup2 and 12.2(18)sxd6. I will stop pim v1 from working and do it on all other 6500 with sup720 using COPP and an ACL with policing. Is there any option to filter pim v1 without a big performance impact ? A. In addition to COPP, we have the option of hardware-based limiters on the CPU.   Q. I have a 7600 with sup 720 in an ISP environment. The CPU reaches 100 every time we have a DDoS attack on an end user. What approach should I take ? A. Refer to Protection against Denial of Services, which contains details to avoid DDoS attacks.   Q. Do we need to specifically check the SP CPU by logging into the SP using the “remote login” command? If the RP was quiet and the SP was busy, would you have to log in to the SP or can you see this without logging in ? A. We can log in remotely to SP, or use the command “remote command switch show process cpu” and “remote command switch show process cpu history”.   Q. Are the drops from input queue related only to the packets that need to be processed switched ? A. That depends on whether it is a layer 3 port or a layer 2 port. For a layer 3 port, input queue – software queue. For a layer 2, input queue – hardware queue.   Q. Is it possible to check forwarding plane utilization for each line card ? A. The CPU on the line card is used for communicating with the Supervisor. This CPU will not be used for maintaining the control plane protocols and will only be used for responding to diagnostic requests from the Supervisor. You can use "attach <module no>" command to log on to the module and check the cpu utilization, using the "show process cpu" command.   Q. Why does the TTL value become 1 during Multicasting on 720 which in turn causes high CPU ? A. It depends on the kind of network with which we are dealing. It becomes 1 in case of loop in the network causing the TTL to go down. This should not be an issue with high CPU; rather high CPU is a symptom of some other issue like a loop in the network. It can also be caused of a bug within an application causing the TTL to go down to a value of 1.   Q. Since each line card has specific forwarding capacity, can we check each line card's forwarding utilization ? A. Yes, the "show platform hardware capacity" command will show the utilization.   Q. How can we reduce the process for SSH on 6500? A. If it is virtual exec, that is used for servicing vty lines, Vty lines are used for logging into the switch. If we are trying to dump a huge output like "show tech", it is expected to see high CPU and is not a matter for concern.   Q. How to troubleshoot CPU spike due to IOS-BASE ? A. High CPU utilization under the IOS-BASE process is for Supervisors running modular IOS. First, find out the PID of the IOS-BASE process from the output of the "show process cpu" command. Following that use the command "show process cpu detail <PID of the IOS-BASE process>" and find out which sub-process under the IOS-BASE process is consuming most of the CPU cycles.   Q. Why does running “show tech” increase the CPU? Is this due to monolithic IOS ? A. Seeing a very small CPU spike while running “show tech” is expected as it is caused by “SSH Process”. It is not a result of IOS.   Q. What is the best way to baseline the traffic in a network in order to implement hardware based limiters or COPP ? A. These are two different things. Hardware rate limiters are available for specific features. If these are not available for the feature we are looking for, we need to fall back to COPP. Hardware rate limiters are best option because of where packets are dropped (right at PFC). In COPP, it is done in software, so the packet needs to go from PFC to software and is dropped in between.   Q. In an ideal situation, what would be the CPU utilization of my switch? A. Ideally, when network protocol has converged, it should be 0% or, at a maximum, 10%.   Q. If I upgrade my Sup2 to Sup 720/Sup 32, can I expect to see the difference in CPU utilization? A. Yes and no. Depending on what caused the CPU utilization, we need to check if upgrade will help or not.   Q. Do we have a CPU on the Line cards running DFCs? Do we see high CPU utilization on the LCs running DFCs as well? A. Yes, we have a CPU on line cards running DFCs. However, that is not used for maintaining control plane protocol, it is used for diagnostic purposes. This CPU is used for responding back to Sup in case Sup is polling towards LC. You might see a CPU utilization there, but it has to do with internal management of LC and system, not network traffic. You should get in touch with TAC in such instances as this can be a bug as well.   Q. Is there any dependency of IOS for capturing process utilization, and, if a process is not identified, will it be seen as spike on CPU utilization? A. If there is CPU utilization due to process switching, you should see some process associated with it. If you do not see a process, it could be a bug in the IOS code.       Questions Related to the Tools Available Q. Can hardware based rate limiters limit to zero, because the customer want to stop the automatic fallback to pim v1 for security reasons? A. Yes, we can make it zero.   Q. Does the Netdr capture would not overload the CPU?­ A. In version 12.2(18)SXF onwards, you do not see any impact on the switch by enabling Netdr capture. On older versions of code, it is just like enabling any other debug, there would be some impact on CPU.   Q. Would setting up a CPU rising threshold capture the SP CPU? A. CPU rising threshold monitors both RP and SP. If you need to monitor only SP CPU, then this EEM script is helpful: event snmp oid 1.3.6event manager applet POLL-SP-CP .1.4.1.9.9.109.1.1.1.1.7.2 get-type exact entry-op gt entry-val "60" poll-interval 10 action 1.1 syslog priority notifications msg "CPU utilization >60% on SP processor, logging data" action 2.1 cli command "enable" action 2.2 cli command "rem comm swi show proc cpu sort | redirect bootflash:SP-CPU"¬ action 2.3 cli command "end” . Q. Can we run Netdr capture on 7600 platform as well? A. Yes, it is the same for 7600 platform.   Q. I can see a lot of UDP packets punting to my CPU. At the same time I can see that “UDP no port” is rising in the “show ip traffic”. Can I configure the 720 not to process unwanted UDP traffic for the servers without the UDP service running? A. First analyze the netdr captured outputs in order to understand why these UDP packets are hitting the CPU. If these packets are to be dropped, use an ACL.   Q. Do we need to be cautious with the "show buffer input-interface" command? A. It is safe to use this command as it just takes a dump of packets in the buffer.   Q. Can we use an ELAM to get specific traffic to the CPU or is this for data plane only? A. Yes, we can use ELAM to find what traffic is going to CPU. Note that this is an internal tool, so you should have TAC assistance to run this tool.   Q. If my top process is “IP Input”, will I have to capture the packages and analyze them on an external system? A. Yes. The IP Input process handles the interrupt/process switched traffic, meaning that the traffic is handled by CPU. As a result, we need to do a packet capture to see what traffic is being processed by CPU.   Q. Can RP/SP inbound span be used on the 7600 or are there limitations? A. Yes, you can use RP/SP inband span on 7600 as well. There are no limitations.   Q. Is there any recommended COPP for 6500s from Cisco? A. No, we do not have any recommended COPP configuration. It totally depends on your network and traffic pattern.   Q. How do I take a capture for the packets punted to the MSFC in case there is a redundant supervisor? A. In this case, only one sup will be in active state. As a result, you will not see high CPU on redundant supervisor. You will see high CPU on active only, so that is what you need to t/s using the tools we discussed.   Q. Will there be an increase in CPU utilization, if we have an EEM script running? A. No. Refer to the Feature Navigator tool in order to check which version supports it. After that version, there is a specific driver used for EEM functionality. As a result, we should not see any high CPU due to EEM running in background.   Q. Do we have a similar span capture for Sup2 running in native mode? A. Yes, but depending on IOS version running, the commands can differ.   Q. I can see many options which are available with netdr capture. What are the different scenarios in which we can use netdr? A. Netdr can always be used when you are trying to capture packets which are bound to or from the CPU. If you are suspecting that the CPU is not sending certain traffic you can configure Netdr in the "Tx" direction and check the output of "show netdr capture" command.The other options available with Netdr is used for filtering the traffic captured in the "Debug netdr capture" command.   Q. How can we determine the direction of the inband span? A. It will be Tx direction.   Related Information Useful Commands for troubleshooting high CPU Troubleshooting with a Netdr Capture on a Sup 720/6500 Troubleshooting high CPU under interrupts Catalyst 6500 Switch high CPU utilization   ... View more

  Introduction: In this document you will learn about Ether-channel load balancing on Cat 6K, 7600, 4500, 3750. Catalyst 6k and 7600:   How it is implemented on this platform?   The way EtherChannel load balancing works is that the switch assigns a hash result from 0-7 based on the configured hash  method ( load balancing algorithm ) for the type of traffic. This hash result is commonly called as Result Bundle Hash (RBH).   As for an example, let us consider that the port-channel algorithm configured on the box is src-dst ip then the source and destination IP of the packet will be sending to the hash algorithm (a complicated 17th degree polynomial) to calculate the RBH. Now each RBH is mapped to one unique physical port in the port-channel, whereas one physical port can be mapped to multiple RBHs (please look at following example for further clarification).   Let us consider the configured LB algorithm is src-mac and the switch is trying to send packets from 3 different src macs a, b & c over the ether channel( 5/1-2).   Now for packets from "a" the hash algorithm computes a RBH of 6, 5 for "b" and 4 for "c".   It is possible that RBH of 5/1 is mapped to RBH of 6 & 4, 5 for 5/2 but one RBH can be mapped to one physical port only. It is not possible that a RBH ( say 3) is mapped to both 5/1 and 5/2.   Things to check/how to check   ------------------------------   1. What is the configured load balancing algorithm?   From the SP "show etherchannel load-balance"   e.g.   6500-20-sp#show etherchannel load-balance   EtherChannel Load-Balancing Configuration:           src-dst-ip <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<           mpls label-ip   EtherChannel Load-Balancing Addresses Used Per-Protocol:   Non-IP: Source XOR Destination MAC address     IPv4: Source XOR Destination IP address     IPv6: Source XOR Destination IP address     MPLS: Label or IP   6500-20-sp#   2. What is the RBH value chosen for the packet from 1.1.1.1 to 2.2.2.2?   From the SP " test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"   Choose the attributes of the above command based on the output of step 1. If the configured load balancing algorithm is src_ip then just give the src-IP of the packet 1.1.1.1. Since in our example the configured load-balancing algorithm is src-dst IP we will feed both 1.1.1.1 and 2.2.2.2 in the above command.   e.g.     6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2   Computed RBH: 0x5   3. How to check which physical port is mapped to what RBH value?   In certain version of IOS the output of test etherchannel command will give you the physical interface chosen; this step (3) is when you do not get the egress interface information from the test etherchannel command output.   For the IOS where the command gives the egress interface the o/p will look like this.   6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2   Computed RBH: 0x5   Would select Gi3/2 of Po1   For other versions go to the RP and type the command "show interface port-channel <num>  etherchannel". Look at the "Load" column o/p corresponding to a physical interface.   Convert the "Load" value into binary (look at the below example)   6500-20#show interface port-channel 1 etherchannel   Port-channel1   (Primary aggregator)   Age of the Port-channel   = 0d:01h:05m:54s   Logical slot/port   = 14/1          Number of ports = 2   HotStandBy port = null   Port state          = Port-channel Ag-Inuse   Protocol            =   LACP   Fast-switchover     = disabled   Ports in the Port-channel:   Index   Load   Port     EC state        No of bits   ------+------+------+------------------+-----------     0     55     Gi3/1    Active    4     1     AA     Gi3/2    Active    4   Here the load value for gi3/1 is "55" and for gi3/2 is "AA".                                       7654  3210 gig3/2 - AA -1010 1010                       |      |                      A       A   gi3/1 -  55  - 0101 0101                    ------   -----                                       |       |                        5      5   for gig3/2 bits 1,3,5 and 7 are set. So RBH value of 1,3,5,and 7 will choose gi3/1.   for gi3/1  bits 0,2,4 and 6 are set. So RBH value of 0,2,4,and 6 will choose gi3/2   From the outputs you can observe that 4bits are set for each of the two interfaces. Hence when there are 2 links in the ether channel then each link has equal probability of getting utilized.   However for 3 links in etherchannel the test etherchannel output will look similar to this.   6500-20#show interface port-channel 1 etherchannel   Port-channel1   (Primary aggregator)   Age of the Port-channel   = 0d:01h:05m:54s   Logical slot/port   = 14/1          Number of ports = 2   HotStandBy port = null   Port state          = Port-channel Ag-Inuse   Protocol            =   LACP   Fast-switchover     = disabled   Ports in the Port-channel:       Index   Load   Port     EC state        No of bits   ------+------+------+------------------+-----------     0     49     Gi3/1    Active    3     1     92     Gi3/2    Active    3     2     24     Gi3/3    Active    2   Here the bit sharing ratio is 3:3:2 hence two links has higher probability of getting utilized as compared to the third link (more in the additional section at the end).   This table lists the ratios of the values that each port accepts, which depends on the number of ports in the EtherChannel:   8 1:1:1:1:1:1:1:1 7 2:1:1:1:1:1:1 6 2:2:1:1:1:1 5 2:2:2:1:1 4 2:2:2:2 3 3:3:2 2 4:4   Note: This table only lists the number of values, which the hash algorithm calculates, that a particular port accepts. You cannot control the port that a particular flow uses. You can only influence the load balance with a frame distribution method that results in the greatest variety.   We also support “per module load balancing” for DFC LC, where you can define LB algorithm per module basis.   http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804   For this implementation we would need to keep in mind that the hash decision is taken on the INGRESS line card. If you have configured ether channel LB for the DFC where the actual physical link in the ether channel exist then your Load balancing might not work as you have desired as ingress LC will decide the egress physical port. By default any LC ( with or without DFC ) will load balance traffic based on the algorithm configured on the PFC. To check the “test etherchannel” command session to the DFC module and then issue the command.   For Cat 4500:   How it is implemented on this platform?   On this platform we use the concept of Agg2PhyPort mapping table. Agg2PhyPort table as the array of 8 elements, each can contain a port number, say for 2 ports a and b then {a,b}: .   Hash function will calculate the index into that array based on the input information: so it will be either 0 or 1 (index is 0 based).   Here’s an example:   Imagine you use 3 links in a bundle (say port 5, 10 and 20), then agg2phyport table would look like:   0: 5   1: 10   2: 20   max-length=3 (number of ports in a bundle)   Now, hash algorithm produces, say 7 (for configured input parameters), then index will be calculated as 7%3=1 and port 10 will be selected.   What to check/how to check?     1. How to check Agg2PhyPort mapping table?   "show platform mapping port" is the command however it is not worth doing it as the output of the command provided in step 4 gives you the egress port all the time.   2. How to check the o/p of hashing algorithm?   Not worth checking because of the above reason. The hash value for the 4500 is calculated via a 'rolling XOR' which is Cisco Confidential.   3. Check the configured load balancing algorithm by using the command "show etherchannel load-balance".   4. Use the command "show platform software etherchannel port-channel 1 map " to find the egress interface.   BGL-4500-12#show platform software etherchannel port-channel 1 map ip 1.1.1.1 2.2.2.2   Map port for Ip 1.1.1.1, 2.2.2.2 is Gi2/1(Po1)   NOTE: Software forwarded traffic will use Gi2/1(Po1)   While using the above command please keep in mind CSCtf75400(registered customer only)   If you hit this bug then unfortunately you have to rely on the sniffer capture to get the actual egress interface,   In K5 based architecture we have actually got rid of unequal load balancing problem when the number of links are 3,5,6 or 7 . As mentioned in the doc that we use 8 bits of hash result to  determine the load balancing, in a scenario where we have 3 physical links in the ether channel 3 bits will be chosen for link 1, 3 for link 2 and 2 for link 3. So the ether channel load balancing probability is ( 3:3:2). However in K5 we use only last 3 bits of the hash result ( for 3 links in EC , 5 bits for 5 links in EC , 6 bits for 6 links in EC and so on ) for 3 links in EC. This way we ensured that all the links in the EC has equal probability of taking the traffic. In K5, in order to improve load-balancing determination and flow distribution we stepped away from the “modulo” approach and load-balancing is based on the pre-programmed hardware mapping table.   For Cisco 3750:     On 3750 we use a similar 8 bit hashing algorithm and hence traffic distribution is more even when number of links in the ether channel is 2 4 or 8 ( please look at the common scenario section for details).   Command to check the egress interface in the port-channel is   " test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"   Common Scenarios:   -----------------------------   Ether channel not load-balancing properly?   To understand the scenario it is important for us to determine all the flows which the etherchannel is handling. Number of flows will depend on the configured load balancing algorithm. Let us take an example.   Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.1 ( mac b.b.b ) with a source tcp port of 50 and destination 2000   Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.2 ( mac c.c.c ) with a source tcp port of 60 and destination 2000.   If configured load balancing algorithm is SRC_MAC   Then no of flows = 1   If configured load balancing algorithm is DST_MAC   Then no of flows = 2   If configured load balancing algorithm is DST_PORT   Then no of flows= 1   and so on.   The ways you can capture the flows are:   -> Sniffer - difficult and hectic.   -> Netflow - relatively easier.   -> External monitoring tool.   Once you have a good idea of the flows then check which flow will take which physical interface. Use the tools discussed above to determine the physical interface.   This step will help you to explain why we see unequal distribution of traffic over the physical interfaces.   Here are the few scenarios which can cause unequal distribution?   1. Let us consider we have two flows and two physical interfaces in the etherchannel. It might be possible that one flow is more talkative than the other.   Now consider i have 5 flows out of which one is super talkative, this flow can overwhelm others. Whichever physical interface this flow is choosing will have relatively higher utilization than the others.   Resolution- flow control the super talker, need to look at from the host side.   2.  One very common problem is that you do not have enough number of flows and out of whatever small number of flows which you have most of them are hashing to the same physical interface.   Resolution- Increase the number of flows. Try changing the hashing algorithm to the one most appropriate to the traffic.   3. When you have 3, 5, 6 or 7 physical links in the ether channel then few links will have higher probability of taking the traffic than the other ( based on number of hashing bit assigned to each physical interface )and hence there is an inherent chance the  traffic will be unequally distributed.   Resolution - use 2, 4 or 8 numbers of links in the ether channel.   Related Information: Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches ... View more