Troubleshooting at the subscriber level

David Damerjian · ‎01-25-2012

In the troubleshooting process, there are times that you know that an issue is noticed or reproducible for a specific subscriber, group of subscribers, type of subscriber, etc. This article provides lists of commands and how to run them as they apply to the subscriber level. Obviously not all of these commands would apply to every situation, but it is good to be aware of all of them so you are prepared to use them when the relevant situation arises.

monitor subscriber

This is probably one of the most well known commands on the chassis. It will display all of a particular subscriber’s control/signaling and payload data for all interfaces, services, protocols, etc.

Considerations in running the command and interpreting output:

- If you simply don’t know the subscriber you want to capture, then capturing by “next-call” may get you the information you need.
- You may need to run the command over and over to capture a subscriber that exemplifies the issue you are trying to troubleshoot, but if the problem is rare, doing so may not be an effective approach.

- If you know the call type (Closed RP, Open RP, EVDO, 1X, LNS), or peer PCF or peer LAC, then you can qualify the next call appropriately to more quickly get a trace of A subscriber exemplifying the issue.
- There are various levels of verbosity. Do not turn on higher levels of verbosity if you do not need to do so, as it makes reading the trace (quickly) more difficult. For PDSN, level 2 is usually enough
- By default, most of the protocols that you would want to view are turned ON by default
- Besides the actual packet data, sometimes you will see CONTROL messages displayed that may explain what action is being taken under the covers – this information is often useful. This includes the call statistics displayed at the end of a call.
- If you need to see all the internal processing for the subscriber (much of which is hard to understand without training to do so), then use the logging monitor config command (discussed later)
- The timestamps displayed are fairly accurate, but, because various facilities are all writing to the screen real-time, you can’t necessarily conclude that the order of packets displayed is the actual order that the packets are being processed in, but it will be close.
- On the ingress side (for PDSN), if you want to view the A11 messaging, then monitor by msid, because the username is not yet known (has not yet been presented) at the beginning of the call and so cannot be displayed
- Turning on L3 User Data, for the most part, will not get you any more information than you would otherwise get, but it will result in duplicates of many packets.
- Certain protocols will result in duplicate packets, for example for Mobile IP, MIP packets will display twice, as PPP and as MIP.
- The output reads much better with a non-proportional font such as courier because the columns line up perfectly, so you may want to do your analysis in your editor with such a font.
- The output that passes through a particular interface should line up with a packet capture on that interface, the difference being that the monitor subscriber output would be a subset because not every single field in an IP packet will be displayed, as it is not necessarily relevant in troubleshooting the protocol in question. For example, most fields from IP headers are not displayed.
- Understand that a lot of the output will be interpreted according to the standard, so instead of displaying an actual integer value, you will see the text representation of the value. Turn on level 3 and/or the hex/ascii dump to see the raw data.
- Not all fragmented packets will be displayed because the Network Processor Unit (NPU) will combine fragments received from the wire before giving them to sessmgr where the monitor subscriber output it generated. Similarly for the outbound direction, fragmenting done by the NPU will not be displayed.
- On a Combo FA/HA chassis, you will only see output for one of the user sessions. For example, if you see radius authentication on the FA, you will not see it on the HA. Use monitor protocol in these situations, if possible, for the specific protocols to be captured.
- When troubleshooting issues between chassis (i.e. FA and HA), you may want to take a trace on both chassis if the problem spans the chassis or if doing so could help eliminate some possibilities.
- packets sent out and received from the egress interface of the Packet Data Network (PDN) (this does not include the egress of an FA for example, because the egress is an FA/HA tunnel), are NOT displayed per the architecture of the system. If a packet comes in the ingress and then a response is sent out the ingress, then you can be sure that the packet made it to its destination and back (including if the destination was the chassis itself). But if no response is sent out the ingress and it was expected, then you need to determine, did it get sent out the egress, and if so, was a response ever received on the egress? Packet sniffers on the respective egress interface and other points in the transport network, including logging at various points including the destination, may be helpful in pinpointing the cause of non-response.
* There is no built-in packet sniffer on the STxx interfaces for customers to access.

When analyzing the output, consider the following:
- are there packets missing (i.e. responses from outside the chassis or requests or forwarding from the chassis)
- can packets be seen going in the opposite direction than is being troubleshot (to confirm at least one direction is working)
- are packets being sent/received at expected intervals realtime and/or per timer considerations
- are packets being sent/received in expected order per the protocol (see caveat earlier about ordering)
- are the various fields in packets containing the correct/expected values
- is the problem consistently happening or randomly

logging monitor

This is actually a global config command versus a runtime exec command used to monitor specific users. There are two advantages to using this command:
- no need to keep a CLI session open because all output is saved to logs
- automatically saves debug level logs for all facilities for the user, including the output that normally would appear with monitor subscriber. Though debug is a lot more than may be needed, you can be sure that no information will be missed, and it is not intrusive to the system as turning on logging would be for various facilities (which would apply to all users)
- output may not be as easy/quick to read as that generated by monitor subscriber.
- in order to capture the details of protocol messages as would be captured by monitor subscriber (logging monitor only saves the first few lines of any protocol message), you'll need to either run a monitor subscriber session as well and later reference the output as needed when analyzing the logging monitor output, OR, turn on full event verbosity for logging (global config command "logging display event-verbosity full"). The latter approach will not require any additional work on your part afterwards because all of the output needed is already together.

logging trace

As opposed to logging monitor, this approach uses exec mode level access, but at the same time also requires the device to be already connected. This is useful if you want to troubleshoot user data (and or further call control events after the point of running the command) as opposed to call setup because the call needs to be already setup in order for this to work (otherwise is reports "No calls match the specified criteria" and achieves nothing). Similar to logging monitor, "show logs" displays all captured data.

-----------------------------------------------

monitor protocol

This monitors all protocol exchange on a chassis for the specified protocol(s) and the output is similar style as monitor subscriber
- This should only be a last resort on a production chassis because of the potential load exerted, dependent on the protocol.
- To get the output for a specific subscriber, it would need to be filtered by identifier-type of information, such as username/msid, callid, etc., which can be done but is subject to error and overlooking data.

active or runtime logging

Captures output for specified facilities at specified level (range error to debug)
- Same issues as with monitor protocol with regards to system load and filtering subscriber output
- Likely requires running/configuring syslog server depending on facilities, level, and timeframe, else data will be overwritten.

----------------------------------------------

For all of the below subscriber commands, you can not only get the information for a particular subscriber, but you can specify a group of subscribers by any number of criteria, such as chassis service they are attached to (PDSN, FA, HA, LAC, LNS, ECS, etc.) or entity that are communicating (peered) with (PCF, FA, HA, LAC, LNS, etc.), PSC/PAC card attached to, connected or idle time left or session time left (greater/less than), amount of data received or sent (greater/less than), associated IP pool name, dormant/active, etc., or even combinations of these. As mentioned earlier, you do not always know of a specific subscriber you want information on, but you do know of a category of subscribers, and this will help you obtain a list of the subscribers in the category, from which you can further narrow/analyze. See the CLI auto-complete help for “show subcribers”. When dealing with a specfic subscriber though, you will need to qualify with the keyword at the end, such as username, imsi, etc.

show subscribers full
Ton of information for the specified subscriber and can be very useful for troubleshooting subscriber issues.

show subscribers aaa-configuration
All aaa related information assigned to a subscriber, regardless of whether aaa was ever accessed or not. Useful to see what the chassis assigned to the subscriber without having to analyze/rely on aaa authentication packets or examine subscriber profiles or make assumptions about the default chassis settings.

show subscribers activity
Graphs activity level of a subscriber

show subscribers data-rate high/low
data-rates for a subscriber or group of subscribers - for example look at thruput for subs in an ip pool

show subscribers debug-info

show active-charging sessions full
very detailed information on enhanced charging for a subscriber

show active-charging firewall statistics

additional information beyond active-charging sessions full

show active-charging flows ip-address

list of all flows by flow id for all sessions connected to the given egress ip address, along with the number of bytes sent in both directions. You first have to use monitor subscriber to see what addresses that a subcriber is trying to access and then confirm whether any packets are received from that address. Detailed information about the flow id of interest can be retrieved with show active-charging flows full flow-id, identifying the proper flow by the MS IP field (IP address of the subscriber which you would know at this point from mon sub output).

show subscribers policy
subscriber's current policies assigned

show [mipfa | mipha] [full | counters]
detailed MIP related information about a subscriber

show ppp [full | counters | summary]
detailed PPP related information about a subscriber

show rp [full | counters | summary]
detailed A11 (RP interface) related information about a subscriber

show l2tp sessions full
detailed l2tp related information about a subscriber

show rsvp counters
detailed rsvp counters for a subscriber

show ims-authorization sessions full
includes the diameter Gx session ID

* Although the commands “show port npu counters” and “show port datalink counters” apply to an entire interface, if you are trying to see if the system is processing data for a particular subscriber out the egress interface (see mention above of limitation of monitor subscriber), and you have control over that subscriber, then you could try to send very large packets through the network, and see if the interface counters increment by the number of packets sent in the short window during which you send them. Being able to do this with confidence in the results requires making sure the counters for the packet size chosen are not normally incrementing very frequently before running the test.

Imported from Starent Networks Knowledgebase Article # 10620 and modified since then