- Cisco Community
- Technology and Support
- Wireless - Mobility
- Wireless - Mobility Knowledge Base
- Understanding and Troubleshooting IP Pools
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
01-25-2012 07:38 AM - edited 07-05-2021 12:13 PM
Includes basics of IP pool configuration and troubleshooting issues related to IP pools, including examining output such as monitor subscriber, logs, SNMP, alarms, thresholds.
IP pools are ranges of IP addresses that are reserved to be doled out to subscribers when they connect to the network. The system can be designed to have all service types (MIP, SIP, LNS, etc) share a given pool, or pools can be unique per service type and subscriber grouping – the flexibility is there to allow for limitless possibilities from simple to complex. This article is not intended for planning and architecting IP pools and contexts for a new (or existing) system – that is best addressed by a Starent network consultant. Rather, this article explains how to view various aspects of ip pools, troubleshoot ip pools when errors are encountered or the system is not behaving as expected. It explains various warnings and error messages encountered that can be seen in monitor subscriber traces, logs, snmp traps, and/or alarms/thresholds. It may touch on related/relevant areas such as licensing, radius authentication, etc.
Basics of IP pools
Pools must be configured in a destination context in the chassis. This implies that there can be more than one destination context in a chassis depending on what services are provided by the chassis as well as how contexts are configured to handle various groups of subscribers. The destination context is either specified in a subscriber profile, via the command “ip context-name <context name>”, or via the attribute “SN1-VPN-NAME” returned during radius authentication. The destination context is important because that is where the IP pools must be defined. For MIP calls attaching to an FA service, the destination context is not relevant. But for SIP calls (connecting via a PDSN service), or MIP calls (connecting via an HA service), or L2TP calls (connecting via an LNS service), the concept of a destination context is critical, because an IP address is being doled out by the chassis in these scenarios.
Pools can be specified as public, static, and private, and it is important to understand the differences:
- A public pool can provide IP addresses for any kind of call request including ones where a pool name is specified. It will not dole out an address for a static subscriber (see below) unless the qualifier “allow-static-allocation” is specified.
- A static pool can only provide IP addresses for requests where a specific IP address is returned by radius via attribute Framed-IP-Address or is specified in a subscriber template via command “ip address <ip address>” – this latter approach would be more for lab testing and not for production because it would require a separate template for every possible static subscriber which is not practical/feasible.
- A private pool can only provide IP addresses for requests where an IP pool name is specified.
IP pools are often grouped allowing for multiple ranges of addresses to be available for the same set of subscribers. This is done with the “group-name” qualifier when defining the pool. At call setup time pool names are either returned by radius via attribute “SN1-IP-Pool-Name“ or “Framed-Pool” (either works) or are specified in a subscriber template via command “ip address pool name <pool name>”
IP pools reside in contexts that have ip interfaces that lead directly to the internet or to a corporate VPN that itself possibly ultimately leads to the internet. The next hop towards the destination can be optionally specified for subscribers in a given IP pool with the qualifier “next-hop-address”. If not, then the route statements in the (destination) context will be used to route to the next hop address. The other direction coming back from the internet to the context can also be a place where things break related to IP pools. There needs to be properly configured routes in the path between the destination context and the internet that point back to the ip pools or else traffic will not make it all the way back to the context.
IP pools can be temprorarily marked out of service for giving out any new IP addresses, which is called busyout. This can be done during troubleshooting if there are issues with a specific pool and you want to not affect any more users, or when making pool changes where you know in advance that you will be removing a pool and want to minimize the number of subscribers impacted. Busy-ing out the pool will not affect existing subscribers, and due to normal call timeouts, with enough advanced notice, you can busy-out a pool allowing remaining subscribers to eventually timeout (and possibly reconnect to a different pool), resulting in the pool replenishing most of the addresses. The busy-out command is also a config command (versus an exec command), but is a separate config line from ip pool configuration. At the context level in which the pool is defined: "busyout ip pool name <ip pool>".
Viewing status of IP pools
To view information about pools, you must run the commands in the context where the pools are defined, otherwise “No IP Pools are configured for context <cpntext name>” will be returned. The basic version of the command is “show ip pool”. For example, for the following pool definition, with one subscriber connected, run the basic command "show ip pool":
ip pool mippool 20.20.20.0 255.255.255.0 public 0
[destination]CSE02# show ip pool
context destination:
+-----Type: (P) - Public (R) - Private (N) - NAT
| (S) - Static (E) - Resource (O) - One-to-One NAT
| (M) - Many-to-One NAT
|
|+----State: (G) - Good (D) - Pending Delete (R)-Resizing
|| (I) - Inactive
||
||++--Priority: 0..10 (Highest (0) .. Lowest (10))
||||
||||+-Busyout: (B) - Busyout configured
|||||
|||||
vvvvv Pool Name Start Address Mask/End Address Used Avail
----- ----------------------- --------------- --------------- ----------------
PG00 mippool 20.20.20.0 255.255.255.0 1 253
Total Pool Count: 1
It is important to be aware of the distinction between the number of subscribers reported (via show subscriber summary or like commands) and the number of used IP addresses reported by this command. For example, in the case of combo chassis that house both FA and HA sessions, there will be two sessions (pdsn-mobile-ip and ha-mobile-ip) for every subscriber that is anchored to both the FA and HA service on the chassis. Another example would be a SIP call (pdsn-simple-ip) extended via a LAC to an LNS (lns-l2tp) that provides the IP address, where the services are all located on the same chassis. Here is an example of a FA/HA combo showing the same IP listed twice:
[destination]CSE02# show subscrber all
+-----Access (M) - pdsn-mobile-ip (H) - ha-mobile-ip
| Type:
vvvvvv CALLID MSID USERNAME IP TIME-IDLE
------ -------- --------------- ---------------------- --------------- ---------
MXCNAM 093d6ae3 0000012345 dave 20.20.20.4 00h06m33s
H.CNAI 0abae326 - dave 20.20.20.4 00h12m29s
Extended information about pools is available and can be useful in troubleshooting.
The verbose qualifier includes among other things, ip pool configuration information that can be overlooked when just starring at a series of (long) ip pool config lines. For the simple example above:
[destination]CSE02# show ip pool verbose
Ungrouped Public Pools:
Pool: mippool 20.20.20.0 255.255.255.0
Pool Status: Good
Type: Public Priority: 0
Pool-Tag: none
Group: VRF:n/a
Used: 1 Free: 253
Hold: 0 Released: 0
Addr-Hold-Timer: - Limit Exceeded: 0
Total Alloc Req: 22 Total Rel Req: 21
Input Label: n/a Output Label: n/a
Network Reachability Detection Server: Disabled
Unicast Gratuitous-ARP Address: Disabled
Nexthop Forwarding Address: Disabled Vlan ID: n/a
Suppress-Switchover-ARPS: Disabled
Send-ICMP-Dest-Unreachable: Disabled
Explicit-route-advertise: Disabled
Include-Network-Broadcast-Address: Disabled
Group Available Threshold: Disabled Clear: Disabled
Pool-Free Threshold: Disabled Clear: Disabled
Pool-Used Threshold: Disabled Clear: Disabled
Pool-Release Threshold: Disabled Clear: Disabled
Pool-Hold Threshold: Disabled Clear: Disabled
Total Explicit Host Routes: 0
Total Pool Count: 1
Here is an example where pools are grouped, and where there is only one pool in this group. Note also the defined group threshold (more on this later).
Group: Group1
Pool: Pool1 80.207.128.0 255.255.192.0
Pool Status: Good
Type: Private Priority: 0
Pool-Tag: none
Group: Group1 VRF:n/a
Used: 4579 Free: 11803
Hold: 0 Released: 0
Addr-Hold-Timer: - Limit Exceeded: 0
Total Alloc Req: 4559431 Total Rel Req: 4554852
Input Label: n/a Output Label: n/a
Network Reachability Detection Server: Disabled
Unicast Gratuitous-ARP Address: Disabled
Nexthop Forwarding Address: 86.174.216.1 Vlan ID: n/a
Suppress-Switchover-ARPS: Disabled
Send-ICMP-Dest-Unreachable: Disabled
Explicit-route-advertise: Disabled
Include-Network-Broadcast-Address: Disabled
Group Available Threshold: 40% Clear: 45%
Pool-Free Threshold: Disabled Clear: Disabled
Pool-Used Threshold: Disabled Clear: Disabled
Pool-Release Threshold: Disabled Clear: Disabled
Pool-Hold Threshold: Disabled Clear: Disabled
Group Summary:
Group Used: 4579
Group Free: 11803
Group Hold: 0
Group Released: 0
Group Effective Alarm Threshold %: 40%
Group Effective Clear Threshold %: 45%
Group Current Usage %: 27.95%
Group Status: Good
There is also a "wide" version of show ip pool that lists additional information such as number of held addresses in the tabular format, which you may find helpful.
vvvvv Pool Name Start Address Mask/End Address Used Hold Avail Rel Free Group Name
----- ------------- --------------- --------------- -------- -------- -------- -------- -------- -----------
PG00 mippool 20.20.20.0 255.255.255.0 1 0 253 0 253
Total Pool Count: 1
You can see a list of all possible IP addresses in a pool and their respective statuses with "show ip pool address ...":
[destination]CSE02# show ip pool address pool-name mippool limit 300
+-------- (B) Busyout
|
|+------- (F)-FREE (U)-USED (H)-HOLD (R)-RELEASE
||
|| Address NAI/MSID Hash Hold Timer Session Start/Disconnect
vv ============== ================ =============== =========================
Pool: mippool
U 20.20.20.4 f2ed59f3c14e495b - Thu Jul 16 08:48:14 2009
F 20.20.20.1 0000000000000000 - -
F 20.20.20.2 0000000000000000 - -
F 20.20.20.3 0000000000000000 - -
F 20.20.20.5 0000000000000000 - -
F 20.20.20.6 0000000000000000 - -
...
F 20.20.20.254 0000000000000000 - -
Troubleshooting
This section covers some possible types of failures related to IP pools as well as possible steps to take.
Problem: Insufficient resources
Mobile IP error code Insufficient Resources
Message Type: 0x03 (Registration Reply)
Code: 0x42 (Insufficient Resources)
The disconnect reason code at the end of a monitor subscriber trace and reported by “show session disconnect-reason” will be MIP-proto-error
Note that for Simple IP unlike for MIP, insufficient resources results in PPP negotiation failing with a PPP Term sent to the mobile, which by itself is not very useful because there are many potential causes for this. But, the disconnect reason for such calls will be reported as “No-IPV4-address-for-subscriber” which is useful.
Insufficient resources is often related to IP pools as follows:
- IP pools have actually run out of IP addresses to give out.
- this would be considered an outage scenario
- if proper capacity planning and testing has been done, then addresses should not just run out suddenly, and one needs to determine why so many addresses are being allocated. Possible considerations:
- some other chassis in the network is down or not giving addresses or transport issues in network preventing other chassis from taking calls and overflow is being sent to this chassis
- configuration changes made to radius or other equipment in network resulting in calls being sent to this chassis
- new call policy feature enabled on another chassis and pointing calls to this one, or to a Foreign Agent which in turn is connecting to this one for addresses
- this chassis is experiencing an issue with some services, so that addresses are being requested from this particular pool (group) when they would not normally be or at least not at such a high rate. For example, maybe MIP calls are failing down to SIP abnormally resulting in SIP pools are being exhausted.
- if an alert-threshold is configured for the pool, then an alarm and SNMP trap should have been triggered to warn of potential pool exhaustion (more on this later)
- IP pool name specified for the subscriber does not exist (applies to both static and non-static users)
- check that the Radius server is properly configured for the subscriber
- check that the correct subscriber template is being assigned for the subscriber
- no static IP pool or public pool with policy “allow-static-allocation” exists that can dole out requested IP address
- check to see if a static ip pool or public pool with policy “allow-static-allocation” exists that contains the ip address requested
- check that the Radius server is configured with the proper ip address that the subscriber is supposed to be programmed for
- check to see if the address has already been doled out to that subscriber or a different subscriber
- show subscriber trace may display the following:
***CONTROL*** 10:35:14:447 Eventid:10083
Sessmgr-17 Failed to allocate static IPv4 address 85.255.191.1 mask 0xffffffff poolname <Static1> for call (errcode=VPN_MSG_STATUS_FAILURE)
- IP pool(s) have been busyout'd (config mode command "busyout ip pool name <ip pool name>") for some reason and so no addresses can be doled out, even though there are many free addresses. One would normally know this but with multiple personnel, locations, and shifts, it's possible that this information has not been communicated to all who should know.
- Note that if the issue has to do with insufficient licenses, the MIP error code will be “0x81 Administratively Prohibited” instead.
Problem: SNMP trap and alarm triggered indicating pools being depleted
- First, this does not necessarily indicate there is a problem, but it is a warning that there could be a problem
- The alert-threshold qualifier for an ip pool allows for the issuing of a warning when the % of ip addresses available in a group of pools or a specific pool have run low. If this alarm is triggering on a regular basis, then first make sure that the setting makes sense – a high (%) value means that it has been programmed to alert even when there are lots of addresses still available (it does NOT mean % used but rather % free, so a lower number points towards exhaustion - this is a common point of confusion)
- If the threshold is set reasonably, then
- it may simply be that the number of subscribers has increased on a consistent basis that it is time to increase the number of IP pool addresses in the pool(s) and/or to add new pools. If this is done, make sure to obtain a license with increased sessions if necessary.
- If you know this not to be the case, and the pool usage continues to grow, then there may be a real problem. See earlier discussion on running out of addresses.
- Example SNMP traps, log messages, and alarms:
ip pool Pool1 80.207.128.0 255.255.192.0 private 0 group-name Group1 alert-threshold group-available 40 clear 45
Wed Jun 24 09:20:01 2009 Internal trap notification 238 (ThreshIPPoolAvail) context destination group Group1 threshold 40 measured value 35
Wed Jun 24 13:40:01 2009 Internal trap notification 239 (ThreshClearIPPoolAvail) context destination group Group1 threshold 45 measured value 46
2009-Jun-24+09:20:01.215 [alarmctrl 65201 info] [8/0/550 <evlogd:0> alarmctrl.c:180] [context: destination, contextID: 3] [software internal system critical-info] Alarm condition: id 50028f41efc10000 (Minor): <76:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.
2009-Jun-24+13:40:01.044 [alarmctrl 65200 info] [8/0/550 <evlogd:0> alarmctrl.c:273] [context: destination, contextID: 3] [software internal system critical-info] Alarm cleared: id 50028f41efc10000: <76:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.
[local]PDSN# show threshold
Threshold operation model: ALARM
Configured thresholds:
Name: available-ip-pool-group
Config Scope: Context[destination]
Threshold: 40%
Clear Threshold: 45%
Active thresholds:
Name: available-ip-pool-group
Config Scope: Context[destination]
Threshold: 40%
Clear Threshold: 45%
Poll Interval: 300Seconds
Next Poll Time: 2009-June-24+10:15:00
Outstanding alarms:
Threshold Name: available-ip-pool-group
Alarm Source: IP-Pool [vpn/pool: destination Group1 ]
Last Measured: 37%
Raise Time: 2009-Jun-24+09:20:01
[local]PDSN# show alarm outstanding
Sev Object Event
--- ---------- ----------------------------------------------------------------
MN VPN 3 <66:available-ip-pool-group> has reached or exceeded the configured threshold <40%>, the measured value is <35%>. It is detected at <IP-Pool [vpn/pool: destination Group1 ]>.
Problem: Failure to create a ranged pool: “Failure: IP Address Pool too big! (max: 13 bits/255.248.0.0)”
- For example, the following attempt will generate this error: [egress]CSE_2(config-ctx)# ip pool pub003 range 12.0.0.0 12.7.255.255
- Ranged pools by definition include every possible address specified in the range, whereas masked pools automatically exclude the network and broadcast address of the specified mask. So, you may be restricted when specifying a pool using a range depending on the size of the range.
Imported from Starent Networks Knowledgebase Article # 10560
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: