Introduction The intent of this document is to explain in a simple manner the cause of the most common types of crashes and typical actions to take for resolving them. False crashes System returned to ROM by reload Cause : the router was reloaded using the ‘reload’ command. Solution : ensure the right persons can execute such command. System returned to ROM by power-on Cause : the router encountered a power outage, was manually power cycled or its power supply became deficient. Solution : verify the status of the power supply with the 'show environment' and the syslogs (a failing power supply will typically be reported) and if failing, replace it. If the power supply works properly, then the cause is one of the two first possibilities. System returned to ROM by abort Cause : the router received a BREAK signal through the console. Solution : verify the Configuration register from the 'show version'. If it is set to a value of 0xA0BC (A,B,C being any value - example: 0x2002), the configuration register needs to be changed to prevent accepting the BREAK signal except at bootup time. To do so, go to configuration mode and change its value to 0xA1BC (typical value being 0x2102) then reload the router to apply this value   Router#conf t Router(config)#config-register ? <0x0-0xFFFF> Config register number Parity crashes : System returned to ROM by processor memory parity error or Crash due to "Cause = 0x20" Cause : 1) If no parity error was seen within the previous month, the crash is due to a transient failure due to cosmic radiation, involving a bit swap in router memory. 2) If there was more than one parity error within the last month, the crash is due to a hardware failure from the route processor. Solution : 1) Monitoring the router for the upcoming month. If no further parity error is seen, no further action is to be taken. If another parity error is seen, replace the route processor (don't forget to replace the possible memory upgrades as well). 2) Replace the route processor directly (don't forget to replace the possible memory upgrades as well). Bus Error Cause : the processor tries to access a memory location that either does not exist (a software error) or does not respond properly (a hardware problem).   You can find the memory location by looking at the address in the  output of the show version command, as shown in this example:     Router#show version Router uptime is 2 days, 21 hours, 30 minutes System restarted by bus error at PC 0x30EE546, address 0xBB4C4   With the address accessed by the router when the bus error occurred,  determine the memory location that the address corresponds to by issuing  the show region command.   Solution : 1) If the address reported by the bus error does not fall within the  ranges displayed in the output of the show region command, the  router was trying to access an address that is not valid. This indicates the router faces a software problem. The first step towards the resolution is to upgrade the IOS software to the last version. If the crash is still seen afterwards, opening a TAC case would be the second step.   2) If the address falls within one of the ranges in the show region command output, it means that the router was accessing a valid memory  address, but the hardware corresponding to that address is not  responding properly. The next action is to replace the problematic hardware, being most of the time the route processor.</p> Illegal Opcode & Sigtrap Exception Cause : IOS software failure Solution : 1st step is to upgrade the IOS software to the last version, verify if the crash still happens and if so, open a TAC case. Watchdog timeout 1) When no crashinfo file is generated and the message *** Watch Dog Timeout *** is seen on the logs, the cause is most probably hardware. The solution is to replace the route processor. 2) When the message "Process aborted on watchdog timeout" is seen on the logs, and the crashinfo mentions a Software forced crash, the cause is an IOS software problem. Software Forced Crash Cause : IOS software failure, typically a corruption happening in the memory Solution : 1st step is to upgrade the IOS software to the last version, verify if the crash still happens and if so, open a TAC case. ... View more

LSMPI - Linux Shared Memory Punt Interface - memory pool is being used to transfer packets/messages from the forwarding processor to the route processor. This memory pool is carved at router initialization into pre-allocated buffers, as opposed to the Processor pool where IOSXE can allocate memory blocks dynamically. ------------------ show memory statistics ------------------                 Head    Total(b)     Used(b)     Free(b)      Lowest(b)  Largest(b) Processor   3008A008   1758579984   306721100   1451858884   1414499484 1406502920 lsmpi_io   98FF71D0     6295088     6294120         968         968      968 It is therefore expected to see this low amount of free memory in lsmpi_io. In fact, it would be more of a concern to see a large amount of free memory for that pool as it would mean IOSXE wasn't able to allocate that memory. Also, e xamining show buffers and the lsmpi_rx and lsmpi_tx secions, we can see that the free buffers counters are 0. Private particle pools: lsmpi_rx buffers, 416 bytes (total 8194, permanent 8194):      0 in free list (8194 min, 8194 max allowed)      8194 hits, 0 misses      8194 max cache size, 0 in cache      3316582 hits in cache, 0 misses in cache lsmpi_tx buffers, 416 bytes (total 4098, permanent 4098):      0 in free list (4098 min, 4098 max allowed)      4098 hits, 0 misses      4098 max cache size, 4089 in cache      1952353 hits in cache, 0 misses in cache If your snmp application is polling this memory or pool statistics, it will generate false positive alerts. It would therefore be better to discontinue polling the lsmpi_io pool or lsmpi buffers. ... View more

The goal of this document is strictly to provide an introduction to mpls traffic engineering in a vulgarized manner and in no way to be considered as technical reference. Introduction Imagine yourself having to go to some place you don't know by car. Nowadays, we would use a GPS system to do so. However, at traffic peak hours, most of the GPS systems are unable to avoid the traffic jams. Why ? Because they take only into account the maximum speed we can reach depending on the roads. Since more speed means less time, they usually choose highways, then national roads and if no other choice, small roads. In an internetwork, the same problem arises as we use the interior gateway protocols (IGPs) to calculate the best path towards a destination. They only select the best theoretical path to the destination without taking into account the possible congestions. Even though IGPs allow several best paths to be used, similarly to having several lanes on highways, they reach a limit at some point. Even EIGRP with its variance feature will always prefer to send more traffic to the highest bandwidth link while other links could be used much more than they are. All in all, IGPs are like GPS systems. They tell you to choose the best theoretical path to your destination but they don't take into consideration the amount of traffic already present on the road/links. Luckily for us, we don't end up as car drops but frustration can increase over time... After observing these same conditions several days in a row, we decide then to modify our itinary by not listening to the GPS when it tells us to take the highway at peak hours. Instead the GPS has to adapt to the smaller roads we decide to take (even though it tries so much to get us back on the highway). After some time and many trials & errors, we end up finding out the least worst road sections to that place at peak hours which make us gain a precious time as opposed to what our beloved GPS ordered us to do. Since IGPs don't allow us to do so in internetworks, traffic engineering (TE) is born. By using a reservation protocol (RSVP), TE verifies first that the bandwidth we want can be achieved by the link the IGP prefers. If the bandwidth can't be achieved, Advantages: Addresses the inherent limitations of IGPs allowing the use of all the links and not only the least cost path to be used Allows setting up a labelled switched path (LSP) without the use of LDP/TDP/MP-BGP Drawbacks: Requires RSVP and IS-IS / OSPF throughout the traffic path(s) Remains level (IS-IS) / area (OSPF) specific in current implementation Topology: In the example above, and with the use of a typical IGP, only the FastEthernet links R1-R3-R4 would be used to route traffic, making the path R1-R2-R4 only good for backup purposes if one of the FastEthernet links is not available. By introducing MPLS TE, we'll see then how to make use of the two paths despite their bandwidth difference by means of dynamic and explicit path, thus achieving destination-based load-balancing. Step-by-step approach: Ensure CEF is enabled on the router, otherwise label distribution will not happen Enable TE support for IS-IS or OSPF IS-IS metric-style wide mpls traffic-eng {level-1 | level-2} mpls  traffic-eng router-id <router-id> OSPF mpls traffic-eng area <area-id> mpls traffic-eng router-id <router-id> Enable mpls traffic-eng tunnels in global configuration mode on all the routers in the path (in the example, enable it on the 4 routers) Enable mpls traffic-eng tunnels and ip rsvp bandwidth on all interfaces in the path Define the explicit path(s) if needed: ip explicit-path {name word | identifier number} enable next-hop <ip address> Define the tunnel with an interface Tunnel X, setting the mode to MPLS TE with tunnel mode mpls traffic-eng, define the tunnel source and destinations with tunnel source Y.Y.Y.Y and tunnel destination Z.Z.Z.Z then specify the tunnel mpls traffic-eng path-option number {dynamic | explicit {name path-name | id path-number}} specifying if you want to use routing protocol learnt path (dynamic) or the path you configured manually (explicit) Advertise the desired destinations through the Tunnel by means of static routing, policy-based routing or dynamic routing ... View more