Community Tech-Talk series is designed to bring technical Experts from Cisco to share their insights on specific topics, selectively chosen based on most-common conversation themes in our technology area on the community from our Support Communities. This Tech-Talk focuses on the concept of Network Virtualization called as EVN. EVN stands for Easy Virtual Networks and as the name suggests they are indeed very easy to understand, implement and deploy. Watch the Tech-Talk Video and checkout the related Blog. I hope you find this video and presentation useful. Please do share your feedback and opinion via  the comments session below.   ... View more

Community Tech-Talk series is designed to bring technical Experts from Cisco to share their insights on specific topics, selectively chosen based on most-common conversation themes in our technology area on the community from our Support Communities. This Tech-Talk focuses on the concept of Network Virtualization called as EVN. EVN stands for Easy Virtual Networks and as the name suggests they are indeed very easy to understand, implement and deploy. Watch the Tech-Talk Video and checkout the related Blog. I hope you find this video and presentation useful. Please do share your feedback and opinion via  the comments session below.   ... View more

Community Tech-Talk series is designed to bring technical Experts from Cisco to share their insights on specific topics, selectively chosen based on most-common conversation themes in our technology area on the community from our Support Communities. Watch the  Tech-Talk Video  and download the pdf copy of presentaion  here .   Virtualization seems to be a buzz word these days. We see software virtualization, host virtualization, data virtualization and lots more. One of the new virtualization techniques that is gaining popularity is Network Virtualization. We keep hearing Software Defined Networking, Cloud Networks and Application Centric Infrastructure but does this mean it is supposed to be a very complex and hard thing to implement or understand? Well let me take you through a concept of Network Virtualization called as EVN. EVN stands for Easy Virtual Networks and as the name suggests they are indeed very easy to understand, implement and deploy.   EVN is a simplified LAN virtualization solution that enables service providers to provide service separation on a shared physical network infrastructure. EVN do not introduce any new protocols and just uses existing technologies like VRF lite, dot1q tagging, trunking etc. The main idea behind is to make deployment of VRF Lite solution easier and optimize it as well.   In Multi-VRF solution we have to define VRFs on all devices and then assign interfaces under VRF. We normally use sub-interfaces and in doing so we lose lot of usable vlans not to mention time required to create these sub-interfaces. IP address planning itself is a huge investment of time for these interfaces. With EVN we try to solve this problems by introduction of concept VNET.   VNET is essentially a normal trunk that carries traffic for different VRF in different dotq vlans. Each VRF that must be transported across the VNET trunks should be configured with a VNET tag. This VNET tag equals a dot1q tag. VNET trunks are not layer 2 interfaces but rather layer 3. When we configure EVN feature then for every VNET defined under VRF we create dot1q subinterfaces automatically created and these are hidden. The configuration of the main interface is inherited by all (hidden) subinterfaces. This automates the process of creating sub-interfaces for each VRF for connectivity towards the core.   We can then run any vrf aware routing protocol over these VNET trunks and establish end to end connectivity for a VRF over a shared network infrastructure. We have options of leaking routes between different VRF just like in MPLS L3 VPN scenario. This feature is available for IPv6 and well as IPv4. It can co-exist with MPLS L3 VPN and normal VRF lite scenarios. It supports deployment of PIM and includes enhanced troubleshooting tools.   Now lets look at how we can configure this feature. We would take a step by step approach using this sample topology:   We have two CE router connected to PE routers at two sites. The two PE routers are connected to each other with a P device in between.   Step 1: Configure VRF Definitions The first would be to define VRF on all the core routers: P, PE1 and PE2. The syntax to create vrf would be: vrf definition [name] vnet tag [2-4094] ! address-family ipv4|ipv6 exit-address-family   Everything is like configuring a normal VRF for MPLS VPN just that instead of specifying RD value you specify a VNET value. This VNET value should be same on all the devices configured for that VRF.   Step 2: Assign CE facing interfaces to VRF Nothing fancy here as well. It is exactly the same as a normal VRF Lite or MPLS L3 VPN scenario. interface GigabitEthernet x/x.y  vrf forwarding [name]   ip address x.x.x.x y.y.y.y   ...   Step 3: Creating VNET Trunks All that is required is a single command under the core facing interface. Once this command is put sub-interfaces for each VRF are automatically created. The dot1q tag used for sub-interfaces is the same that is defined under the VRF. The IP address is inherited from the main interface. These sub-interfaces are hidden. interface GigabitEthernetx/x   vnet trunk   ip address x.x.x.x y.y.y.y   ... Few things to keep in mind are: 1. VNET trunks cannot work if manual sub-interfaces are also present on that interface. 2. If you want to see hidden sub-interfaces then use "show derived-config" command 3. You could use physical as well as port-channel interface for VNET configuration   Step 4: Creating Routing Protocol Instance It is the same like multi VRF lite scenario. You have to configure routing protocol instance per VRF. You may choose any VRF aware routing protocol for this purpose. The only thing to keep in mind is to also include automatically created sub-interfaces as a part of the routing protocol.   Step 5: Configuring P routers There might be routers in your core which are acting as P routers and do not have any CE routers connected to them. Since this is a VRF Lite scenario, you need to configure these routers as well. You would have to configure VRF definition (Step 1), Define VNET trunks (Step 3) and configure routing protocol instance (Step 4). With EVN every time you create a new VRF you do not need to create any sub-interfaces facing the core and this is the real power and ease which you can get with EVN.   There are some advanced features as well that you can configure with EVN. For example:   Route Replication:  Leaking routes from one VRF to the other just like in MPLS L3 VPN scenario is easily possible with EVN and without the the need of BGP. Other good part is that only original route is saved and leaked routes are references to the original route. This way we save on memory.   vrf definition VRF-X   address-family ipv4|ipv6    route-replicate from vrf VRF-Y unicast|multicast     [route-origin] [route-map [name]]   VNET customization: You have options available to:   1. Change VNET tag per VNET interface. 2. By default all routing protocol attributes are inherited from main interface, if you wish you can change that per VRF. 3. You also have an option to limit the VNET tags on a VNET trunk   Watch the Tech-Talk Video and download the pdf copy of presentaion here. I hope you find this blog, video and presentation useful. Please do share your feedback and opinion via the comments session below. ... View more

Community Tech-Talk series is designed to bring technical Experts from Cisco to share their insights on specific topics, selectively chosen based on most-common conversation themes in our technology area on the community from our Support Communities. This Tech-Talk focuses on the concept of Network Virtualization called as EVN. EVN stands for Easy Virtual Networks and as the name suggests they are indeed very easy to understand, implement and deploy. Read the Tech-Talk Blog and download the pdf copy of the presentation slides here. I hope you find this video and presentation useful. Please do share your feedback and opinion via  the comments session below. ... View more

Community Tech-Talk series is designed to discuss and share insights on specific topics, selectively chosen based on most-common conversation themes happening in community from our technology area.   Objectives:   What could possibly cause multicast to be S/W switched Troubleshooting approach Tools that help Questions from the Cisco Support Community   Checkout the Blog & presentation slides ... View more

Community Tech-Talk series is designed to bring technical  Experts from Cisco to share their insights on specific topics,  selectively chosen based on most-common conversation themes on our  technology area on community from our technology communities. Session Objectives: What could possibly cause multicast to be S/W switched Troubleshooting approach Tools that help Questions from the Cisco Support Community Watch the Tech-Talk and checkout the related Blog I hope this has been an informative session and proves useful for  troubleshooting high cpu due to mulitcast. Please do share your feedback and opinion via the  comments session below. Referenced links: Troubleshooting high CPU on 4500 devices Troubleshooting high CPU on 3560/3750/ME3XXX Platforms Configuring Embedded Packet Capture IP Multicast Troubleshooting Guide Basic Multicast Troubleshooting Tools ... View more

Tech Talk series is a great platform to share knowledge on specific topics like troubleshooting and advanced features which is normally hard  to find. Being a TAC engineer with both Routing as well Switching team I  come across high CPU issues almost on a daily basis. The most common  reason that I find is that high CPU is because of multicast traffic. So I  though why not use this amazing platform to share some of the insights  regarding the troubleshooting process. Common problems specfic to multicast traffic that cause high CPU Lets first see few of the common situations which would let multicast traffic not be CEF (Cisco Express Forwarding) switched: Note:  In the Video as well as the blog software switched means packets that  are NOT switched by CEF and hardware switched means packets that are  either swithced by software CEF for software switched platforms or  swithced by hardware CEF (not software CEF) in hardware switched  platforms.  1. CEF is disabled: Title says it all, if CEF is disabled then we would have to switch  packets in software. This is true for unicast as well as multicast  traffic. 2. Presence of "ip igmp join-group" : When we do not have an option to receive IGMP reports from clients or  if we wish to ensure multicast continues to flow through an interface  even if there is no receiver present then we need to statically  configure join for a multicast group on the router for that interface. There are two interface levels commands to do so : A) "ip igmp join-group <group_address>"  makes the multicast to go out of the interface as well as send a copy  of multicast traffic to CPU. So every multicast packet coming in (for  the multicast group configured) would be sent to the CPU. B) "ip igmp static-group <group_address>" makes the multicast  to go out of the interface and not send a copy to CPU. So  if you want to configure statically a multicast group in production  network then option B is our choice. We normally use "join-group" during  pre-production to ensure our multicast is fine due to the fact that  multicast traffic goes to the CPU and thus the router can take some  action like for example send an ICMP reply for an ICMP request send to a  multicast group.  3. Traffic Failing RPF (Reverse Path Forwarding) check: Multicast traffic that fails RPF check is always sent to CPU. RPF is a  mechanism to ensure there is no loop for multicast traffic in the  network. We only forward if we receive multicast traffic on RPF  interface. 4. PIM registration Process: When  a source needs to register with RP (Rendezvous Point) the First Hop  router encapsulates the multicast packet into a unicast packet. This  process is completely done in software so if registration process is not  getting completed then all the multicast traffic would be software  switched on first hop router. 5. Traffic to reserved multicast groups: Multicast traffic destined to 224.0.0.1 - 224.0.0.255, 224.0.1.39 and 224.0.0.1.40 is always software switched. 6. TTL = 1 packets: Packets having TTL=1 and needs multicast routing are always sent to CPU. 7. Fragmentation: If packet needs to be fragmented when it needs to be sent out then we  need to sent it to CPU as fragmentation cannot be done in hardware. 8. Platform Limitations:  Before designing multicast we must ensure we meet all the criteria for  traffic to be switched in hardware for that particular platform. Troubleshooting Approach and Useful Tools Now lets look at the troubleshooting approach we would follow: 1.  First we need to determine what kind of packets are hitting the CPU.  There are majorly two ways to do that: First is to sniff the CPU by  connecting a PC running Wireshark or Ethereal to the problem device.  However most of the times this is not possible as the device is at  remote location. The other way to do so is run some platform specific  commands: A) 7600/6500 platforms: "Netdr Capture"   This is an internal buffer which can capture up to 4096 packets that  are going to the CPU. It is safe to run in high CPU sitations: i)   To enable the capture: "debug netdr capture rx" ii)  To display the packets captured: "show netdr capture" iii) To clear the capture buffer "debug netdr clear" iv) To stop the capture "undebug netdr capture" B) 4500 platforms: "CPU packet dump utility" Details can be found here: Troubleshooting high CPU on 4500 devices C) 3560/3750/ME3XXX platforms: "CPU receive queue dump utility"  Please take care while using this as this command floods the console with lot of data. Details are available here: Troubleshooting high CPU on 3560/3750/ME3XXX Platforms D)  For platforms like ISR/7200 packets are swithed using software and  there is not way to sniff or dump the packets going to CPU. We could  only see packets in the input buffers of an interface using the command "  show buffers input-interface <> packet" but since packets  dequeued very quickly for process we cannot see all the packets. Other  way to dump the packets is to use Embedded Packet Capture (EPC) utility  if you are running release 12.4(20)T. Information on EPC can be found here: Configuring Embedded Packet Capture 2.  Next we need to analyze what we captured and see if we could find what  is causing CPU to go high. If we see that it is multicast packets then  what we need to do is zero in on a multicast group which we see hitting  the CPU the most. If we cannot zero in on any then just choose one based  on your best judgement. We would call this as the problem group and  probably if we solve issue for this group then may be we could apply the  same solution for the other groups. Tools that can  help us with this job depend on what kind of captures we have, If we  have wireshark or ethereal captures then we can use filter expressions  to narrow down the problem group. However if we have outputs from  inbuilt CPU sniffer captures then we cannot use softwares like Wireshark  . The way to go about this would be to use Linux/Unix commands like "grep" in combination with "cut", "uniq" and "sort" keywords. For  example let us say we have a netdr capture (in a file netdr.txt) and we  want to find out number packets received for each destination IP then  this is the command I would execute from the directory containing  "netdr.txt" file: grep 'ttl' "netdr.txt" | cut -d, -f6 | sort | uniq -c What this command would do is parse the complete netdr.txt and select the lines containing "ttl". These would be the same lines having Destination IP. Next we do a "cut" operation which would extract only destination IP address from the line. Now we do a "sort" so that all same destination IP come together and afterwards we could use the "uniq" keyword to count consecutive same destination IP. More details can be found about these keywords by referring manual of each. 3.  Once we have our problem group, let us analyze the packets for this  problem group and see if packet itself requires special handling and  this the reason it is going to the CPU. We need to check: A) If TTL value of packet is 1. B) If the length of packet is more than MTU of configured on the interfaces. C) If destination IP is in reserved range that is 224.0.0.1 - 224.0.0.225 or 224.0.1.39 - 224.0.0.40 D)  if any IP options are present. If they are present then packet would be  handled by CPU as IP options cannot be processed in hardware. 4.  If we do not find anything wrong with the packet itself then we need to  check if there is something wrong with the network or configuration. We  would follow the following sub steps: A) Check if we have "ip igmp join-group <problem_group>" present in the config. if it is present then change it to "ip igmp static-group <problem_group>". B) Check which PIM flavor you are running for that multicast group. Depending on the flavor your multicast tree would be formed. C) Check "show ip mroute <problem_group>" to see if inbound and outbound interfaces are correctly listed and are  in accordance with multicast tree that is supposed to be built. D) Check "show ip mroute <problem_group> count" to see if RPF is failing. If it is then this is the reason of high CPU  and we might need to see why multicast traffic is not coming through RPF  interface. It is quiet possible that PIM is not enabled on RPF  interface or there are some "static mroutes" wrongly configured. There  might be other reasons and we would need to refer PIM troubleshooting: IP Multicast Troubleshooting Guide E) If we see "Registering" flag in "show ip mroute" then it means there is some problem in registration process and we need  to check why it is not getting completed. Probably we do not have route  to RP on first hop router or route to source on RP. We might also have a  problem with (S,G) tree between First Hop router and RP. We would need  to go hop by hop starting from RP to find what is wrong with multicast  tree. 5) We could also run some debug  and show commands related to multicast which could help us figure out  what is going wrong. Please refer this link for such commands and debug  outputs: Basic Multicast Troubleshooting Tools 6)  We must by now should have found the problem but if not we might need  to check few other things like if IGMP snooping is disabled for a vlan  and if we have a SVI with IP address for that vlan then packet would be  sent to CPU. The reason is now all multicast packets would be flooded in  a VLAN at layer 2 and thus would be sent to SVI interface also. Since  ownership of SVI lies with CPU packet is sent to CPU. We could see if we  have correct CEF entry and if CEF entry is not present then traffic  would be sent to CPU. There might be some platform limitation which we  could find in configuration guide for that platform. Watch the Tech-Talk and checkout the presentation slides I  hope this has been an informative session and proves useful for troubleshooting multicast high CPU situations. Please do share your feedback and opinion via the  comments session below. Thank you for watching! ... View more