Overview
In the previous note, "Why Frame Relay?", we had a long journey trying to understand the subtleties of Frame Relay technology as well as its configuration.
For any troubleshooting attempt to be successful, you need to have a good understanding of the technology, so you may want to spend some time reading the first note about Frame Relay, if you haven’t done so yet.
People get scared of Frame Relay not because troubleshooting is complex, but because they lack the theoretical knowledge. Actually, troubleshooting can be considered an easy task since there are only a few commands to be used and they give you all the information needed to solve any problem.
Most Common Troubleshooting Commands for Frame Relay
Besides the general troubleshooting command “show interface“, you will most commonly use these two commands when dealing with Frame Relay problems:
- “show frame-relay pvc” –displays statistics about the permanent virtual circuits (PVCs)
- “show frame-relay map” –displays the current Frame Relay map entries
These two commands (together with “show interface“) should be your starting point in your Frame Relay troubleshooting journey.Of course, there will be other commands used for troubleshooting, but they would only be used in some specific situations, as we will see later in this note.Before going into details, let’s see what kind of information we get with these two commands. 
Run without any arguments, the command shows, in the beginning, a summary of all the PVCs and their statuses – active, inactive, deleted or static – as well as their usage – local, switched, unused. The rest of the output contains detailed information about each DLCI, such as input and output packets and bytes, drops, FECN/BECN/DE packets, broadcasts, 5 minute input and output rates.The hard part is to interpret the PVC status and DLCI usage. Most of the time, you are not really interested in the packet or byte count, so a simplified version of this command can be enough: 
show frame-relay map 
Important information is revealed with this command, such as:
- Interface and its status (for example: serial0/0 – up)
- Destination IP address (for example: 192.168.1.2) for the multi-access (sub)interfaces
- DLCI used to reach that destination
- How the information was learned – statically or dynamically
- PVC status – active, deleted, etc.
With all this info in our toolbox, let’s dive into some scenarios and also discover some other useful commands.In the beginning, we will examine two of the most common problems with Frame Relay: PVC statusinactive and deleted. The target is to have an active status for the PVCs; after solving that, you can move to troubleshooting higher level problems. PVC Status InactiveThe router learns all the DLCI provisioned by the service provider Frame Relay switch through LMI and then it tries to use them, either on the main physical interface (by default) or on some other sub-interface according to the configuration. The “inactive” status means that there is a problem with the end-to-end connectivity and most of the time, it indicates that the problem is not local, but rather on the Frame Relay switch or on the DTE router on the other end:
In the example above, there is a missing command on the Frame Relay switch on the interface connected to R1 that has this result:
- R1 does not learn any DLCI from the Frame Relay switch.
- R2 learns DLCI 201 but its status is inactive due to end-to-end connectivity problems.
- FR-SW (Frame Relay switch) also displays an “inactive” status for output DLCI 102 when using command “show frame-relay route“.
Note: Instead of using pairs of “frame-relay route” commands under each interface on the Frame Relay switch connected to DTE routers, you can use the global command “connect R1-R2 serial0/0 102 serial0/1 201“.In the next example, the status of inactive is reported on R2 because of a configuration problem on the other DTE router R1:
In this example, the LMI type is changed from default to Q933A, which causes a communication problem between R1 and the switch. As you can see, there are no DLCI learned on that interface since there is a LMI mismatch between them. This problem on R1 makes R2 report the PVC asinactive.As a troubleshooting rule: whenever you see PVC status inactive, the problem is most likely on the other router or on the Frame Relay switch. PVC Status DeletedAnother common issue is PVC status deleted and it represents a local problem.Whenever you manually configure a DLCI on a (sub-) interface – via either of these commands: “frame-relay interface-dlci 222” or “frame-relay map ip ip_address> 223” – the router expects to learn these DLCIs (in this example: DLCI 222 and DLCI 223) also from the Frame Relay switch via LMI. If there is a mismatch between the locally configured DLCI and the one(s) learnt via LMI, the local DLCI is marked as DELETED. 
In the above output, the only DLCI learnt from the switch via LMI is DLCI 201.Most of the time, the problem of PVC status DELETED is caused by typos in the configuration on the local router. In order to fix them, you have to match the list of DLCIs received from the switch to the list of locally configured DLCIs. Of course, if you know for sure that the locally configured DLCI numbers are correct, then you need to contact the service provider to provision them on the Frame Relay switch.Another possible cause for this problem is the lack of LMI communication between router and switch.As a troubleshooting rule: if you see PVC status deleted, most likely the problem is local on the router. DLCI Usage UnusedAs the word implies, whenever a DLCI learnt from the switch is not usable by the router, it will mark it as unused. This is usually a configuration problem on the local router: 
In the example shown above, the DLCI is unused because there’s no IP address configured on the interface. DLCI Usage SwitchedIf you see that the usage of a DLCI is SWITCHED, this will be a clear indication that the device is performing Frame Relay switching, so it acts as a FR switch. Below is an output taken from a Cisco router that is configured as a FR switch:
Notice another very useful troubleshooting command applicable to the Frame Relay switches: “show frame-relay route“. PVC Status StaticYou can configure back-to-back Frame Relay between two routers connected directly via a serial link without any Frame Relay switch in between. Since there is no switch, there are no PVCs advertised through LMI so you disable it with the command “no keepalives” and the PVC status will be “static“, meaning that it is not influenced by LMI.
Remember that the static status means that the LMI is disabled, so if you see it on an interface connected to a switch (that talks LMI) you should remove the “no keepalive” command.Mismatched Frame Relay Encapsulation Whenever a non-Cisco device is involved in a Frame Relay setup, you have to change the default “cisco” Frame Relay encapsulation type to “ietf“.
The encapsulation mismatch problems can be detected using the command “show frame-relay map” as indicated in the above output.NOTE: With Cisco, the configured encapsulation type is used to encapsulate the packets sent onto the virtual circuit, but the received packets can be de-capsulated no matter if they are “cisco” or “ietf“. As a result, if the endpoints are two Cisco routers and both are configured with different encapsulation types (as in the above example), there is no connectivity problem. Of course, if one router is Cisco and the other is non-Cisco, the encapsulation mismatch will cause connectivity problems.Mismatched LMI Type Local Management Interface (LMI) provides signaling and diagnostics between the DTE router and the Frame Relay switch. The DTE router will learn all the DLCIs on that particular link from these LMI messages. Cisco routers support 3 standards for LMI type – cisco, ansi and q933a – that are not compatible between each other, so you have to make sure that local communication between the DTE router and the DCE switch uses the same standard.If the LMI type is different, the line protocol of the interface will be down and the best command to troubleshoot this scenario is “show frame-relay lmi” which will show you the increasing number of LMI status timeouts, as displayed in the output below:
As a side note, remember that the encapsulation type needs to match per end-to-end virtual circuit while the LMI type needs to match per local DTE-DCE pair, as shown below: 
Keepalives Not Sent As explained above in the “PVC Status Static” section, the LMI can be disabled with the command “no keepalives“. Connectivity problems will occur if this command is improperly used such as on an interface connected to a Frame Relay switch, as in the example below: 
Without LMI, there will be no DLCI learnt from the switch, so the “show frame-relay lmi|pvc|map” commands will return an empty output.Note that mismatched keepalive timers between router and switch can also cause connectivity problems. By default this timer is 10 seconds and it is recommended to match on each side.*When you deal with a Frame Relay problem, you will have to make sure that the status of the PVC is Active and DLCI Usage is Local. Once this is solved, according to the above troubleshooting sections, you can move forward if the problem persists. In the next sections we will cover the problems that are not visible within the PVC status. Incorrect DLCI Association to (Sub)-Interfaces By default, all DLCI learnt from the switch via the LMI messages are assigned to the main physical interface. The PVC status will be active because the end-to-end communication over the virtual circuit is ok – you will see that InARP messages are successfully exchanged between the routers but the problem may exist at the layer 3 level/IP addressing.In the example below, the DLCI 301 is learnt on the interface Serial0/0 and is automatically assigned with it. The IP addressing between R1 and R3 is: R1 = 192.168.1.1/24 and R3 = 192.168.1.3/24, but this address on R3 is configured on a separate multipoint sub-interface (serial0/0.13):
You can see that PVC status is active, as there is a dynamic mapping for IP 192.168.1.1 (R1) to DLCI 301, but this DLCI is incorrectly used on main Serial0/0 interface (that has an IP address of 3.3.3.3/24) instead of the s0/0.13 sub-interface.To correct this problem, the recommended solution is to move DLCI 301 to the sub-interface with the command “frame-relay interface-dlci 301” under the sub-interface configuration: 
Another solution would be to configure a static mapping for IP 192.168.1.1 to DLCI 103 on the sub-interface: 
Note that you will not be able to configure “frame-relay map ip 192.168.1.1 301” on the sub-interface as long as DLCI 301 is already used on main interface. The router would only report the following error: R3(config)#int s0/0.13R3(config-subif)#frame-relay map ip 192.168.1.1 301 broadcast%PVC already assigned to interface Serial0/0 You will have to make sure that the association of DLCI 301 to the main interface is cleared before using the above configuration – for example, shutting down the interface ser0/0. Dynamic InARP Mapping for Different Subnet The moment you configure an IP address on a Frame Relay interface, Inverse ARP (InARP) messages are sent over all DLCIs configured and/or learnt on that interface. The router will notcheck if the destination IP address from the received InARP messages is in the same subnet as the IP address of the local interface.
In this example, dynamic InARP mapping is done for IP 3.3.3.3 to DLCI 103 on interface Serial0/0 even though the subnet configured on the interface is different.To solve this, you have several options:
- Move the incorrect DLCI to a separate sub-interface that does not have an IP address configured. InARP messages are sent for every IP/DLCI pair so without the IP address, no InARP messages are sent.
- Disable Inverse ARP for DLCI 301. Note that disabling InARP (command “no frame-relay inverse-arp IP 103“) will only disable sending InARP requests but it does not stop from replying to messages received on DLCI 301.
*After you checked that the interface is UP, line protocol is UP, PVC status is Active, DLCI usage is not unused, the DLCI is assigned to the correct (sub) interface and the mapping is correct, you may still have problems with the routing protocols running over these Frame Relay links.Have a look at the next sections to understand the common problems you may face. For this purpose, we will use the following diagram consisting of 3 routers with R1 used as the HUB and R2 and R3 as SPOKES: 
Missing Broadcast keyword As indicated in the first note, Frame Relay is a non-broadcast technology so in order to simulate the broadcasting communication, a technique called “pseudo broadcast” is used. This is done by sending multiple copies of the same packet on the same physical link.We all know that most of the routing protocols use broadcast or multicast to establish adjacencies between routers.In the following example, EIGRP is enabled on all interfaces on both R1 and R2, and the two routers can reach each other but they cannot establish an EIGRP adjacency:
The easiest way to troubleshoot such issues is to perform a ping to a broadcast address: 
As you can see, you can ping the unicast address but not the broadcast one. Adding the “broadcast” keyword will bring up the EIGRP session: 
This time we have the EIGRP peering up and receiving successful replies when pinging the broadcast address. Also, remember that when configuring multiple mappings for the same DLCI, use the broadcast keyword only once!Missing Required Mapping in a Hub-and-Spoke TopologyHub-and-spoke and partial-mesh topologies create an environment for a lot of mistakes and sometimes the troubleshooting may not be so straight forward.The problems start from the fact that the spokes don’t have direct layer 2 connectivity though they may have IP addresses in the same subnet. In order to have reachability between the spokes, you need mapping between them using the DLCI towards the hub:
Split HorizonAnother problem in a hub-and-spoke scenario is split horizon. In case of distance-vector routing protocols, the hub will not re-advertise a route received from a spoke (R3 in our example) back on the same interface to other spokes (R2 in our scenario): 
In order to receive the routes from the other spokes, you have to disable split horizon for EIGRP under the multipoint sub-interface, as shown above.At the end of this note, let’s make a summary of the most used commands for troubleshooting Frame Relay problems:
- show interface
- show frame-relay pvc
- show frame-relay map
- show frame-relay lmi
- debug frame-relay packet
- debug frame-relay lmi
Two very good documents on Frame Relay troubleshooting can be found here:
