Backup ISDN usage

[priedman1]

Hello Everyone,

I just started a new position and I was informed that one of our US sites received an excessively high bill from the ISDN service provider for all of their usage. The site is configured with a T1 as the primary connection (connected to a 2600 series router) and the ISDN is the backup (connected to an 800 series router). HSRP is configured on each router.

Can someone help me troubleshoot why this ISDN line is being used so frequently? The provider says the T1 hasn't been having any problems so why should the ISDN be used?

Thanks in advance for the help.

[ankurbhasin]

Hi Predman1,

There can be few reasons why isdn line is up and your company have to pay money for that even when your t1 line is up.

As you stated that isdn is configured on another router you must have connected your 2600 router with 800 via ethernet and you must be having default route towards the isdn router now think of the situation when some route is not available on your main t1 router it will throw the traffic to the isdn router and it will come up even when your t1 line is up.

The best way to troubleshoot is check "sh dialer" when isdn is up and check what is the source and destination ip which is triggering the isdn and then check whether that desination is available in our routing table in 2600 router or not.

Then it can be many situations when the routing table does not have the route in routin table and it throw the traffic to isdn router that time you need to have a static route with low admin dist pointing towards the next hop primary router.

Post you config for isdn router. We just need to add one more static route with low admin dist in your isdn router and that will solve your issue.

Regards,

Ankur

[priedman1]

Thank you all for your assistance. It's greatly appreciated. My boss informed me that things on the ISDN bill were fine a month ago but this month, we got hit hard.

Here are is the ISDN router config. Hope it helps.

ISDN backup router config:

version 12.2

no service pad

service timestamps debug uptime

service timestamps log uptime

service password-encryption

!

hostname

!

boot system flash:c800-y6-mw.122-29.bin

aaa new-model

enable secret

!

username password

ip subnet-zero

!

isdn switch-type basic-ni

!

interface Ethernet0

ip address 172.31.26.3 255.255.255.0

standby ip 172.31.26.1

!

interface BRI0

ip address 192.168.25.26 255.255.255.248

encapsulation ppp

dialer map ip 192.168.25.25 name nusrr003 broadcast 16319528981

dialer map ip 192.168.25.25 name nusrr003 broadcast 16319525672

dialer hold-queue 10

dialer-group 1

isdn switch-type basic-ni

isdn spid1 92537347340101 3734734

isdn spid2 92537347350101 3734735

ppp authentication chap

ppp multilink

!

no ip http server

ip classless

ip route 0.0.0.0 x.x.x.x.168.25.25 200

ip route 10.1.16.7 255.255.255.255 172.31.26.2

ip route 10.11.1.63 255.255.255.255 172.31.26.2

ip route 141.x.x.x.255.255.255 172.31.26.1 2

!

logging 141.x.x.149

logging 10.1.1.203

logging 10.1.2.126

access-list 99 permit 10.14.14.9

access-list 99 permit 10.1.3.2

access-list 99 permit 10.1.3.3

access-list 99 permit 10.11.1.63

access-list 99 permit 10.11.2.65

access-list 99 permit 10.33.1.100

access-list 99 permit 141.x.x.102

access-list 99 permit 10.1.2.126

access-list 99 permit 172.31.1.17

access-list 99 permit 10.1.1.203

access-list 99 deny any

access-list 100 deny ip any host 255.255.255.255

access-list 100 permit ip any any

access-list 100 permit icmp any any

dialer-list 1 protocol ip list 100

snmp-server community RO

snmp-server community RW

snmp-server community RW 99

snmp-server community RO 99

snmp-server chassis-id

snmp-server enable traps tty

snmp-server host 10.1.1.203

snmp-server host 10.1.2.126

snmp-server host 10.1.3.3

snmp-server host 10.14.14.9 hefe5

tacacs-server host 10.11.1.63

tacacs-server directed-request

!

line con 0

password 7

login authentication linecon

stopbits 1

line vty 0 4

password 7

login authentication linevty

!

end

[Richard Burts]

Pete

It is interesting to know that last month the ISDN bill was fine and this month it got hit hard. It leads to some obvious questions:

- were any changes made in router configs during the month?

- were there events where the backup router would have become primary in the HSRP?

It might be nice to know a little more detail about how the HSRP is configured. In particular I would be interested in finding if the primary router tracks its outbound interface (I am assuming probably so) and whether the primary router is configured with preempt (I sure do hope so). I would also be helpful to know if there was anything that happened during the month where the Ethernet of the primary router might have gone down.

The reason for wondering about HSRP is that if any of the PCs sent their packets to the backup router then the backup would have dialed to send the packets and started accumulating the ISDN bill.

I guess there is also a possible question about whether any other device could have dialed your backup router.

I do not see anything in the config for any dynamic routing. So I assume that all the routing info are the static routes. I do notice that the default route is configured as a floating static with Administrative Distance of 200. I wonder why it was done this way since there does not appear to be any other way that the router could have learned a default route.

I also wonder about this static route:

ip route 141.130.174.149 255.255.255.255 172.31.26.1 2

I do not understand why the next hop was configured as .1 which is the shared HSRP address instead of being configured as .2 which is the other router and is how the other static routes are configured. Given that 141.130.174.149 is the address of the upstream syslog server, if the .1 address was not reachable, or was on the backup router then any syslog messages would have been interesting traffic and routed through the ISDN (which might generate a good amount of the bill).

HTH

Rick

[priedman1]

Hi Rick,

THanks for your help. I am not sure why the static route is configured to point to the standby address since I inherited this environment. Your suggestion makes more sense.

When I checked the running configs, I don't see tracking of the outbound interface for HSRP on the main router. Just the stanby IP, priority, and preempt. On the ISDN router, justr the standby IP address is configured.

I was thinking HSRP might be at play here but am having trouble confirming my suspicions. Curious why all calls are only 30 seconds long. Sounds like only enough time to dial/set up the call, and then tear it down. Just my 2 cents.

Thanks again. Much appreciated.

Pete

[ankurbhasin]

Hi Pri,

Now think of a scenario if any route is not present in your routing table on 2600 router what will happen ?

It will throw the traffic to the isdn router and because you have a static route configured on isdn router "ip route 0.0.0.0 0.0.0.0 192.168.25.25 200" it will activate the isdn.

So we need to play with one more static route with less admin dist.

It should be like this "ip route 0.0.0.0 0.0.0.0 195"

Now even if any traffic comes which your primary router does not know it will throw it to isdn router and isdn router will again check for static router in routing tabel and will check or lower admin dist which is 195 and it will again throw the traffic to the primry router and in this way isdn will not be activated and when the primary link goes down the static route with less admin dist will be removed from the routing table and your static router with admin dist 200 will activate the isdn.

Please check the "sh dialer" it will lest us know which destination ip is activating the isdn router and whether that destination ip is present in routing table for primary router.

Also please post the ip adress of the serial link on primary router.

I think even snmp traps can active the isdn link if generated by isdn router. Please check "sh dialer" and confirm the ip adresses.

HTH

Ankur

[priedman1]

Hello Everyone,

Here is some more information as per your requests. Thanks again for all of the help. As far as I know, nothing was changed with the router configuration. This past month, the bill was huge and all calls were only 30 seconds long and occurred at multiple points throught the day on pretty much every day of the month.

The IP address of the main router's serial interface is 12.119.241.82

-----

ISDN router "show ip route"

Gateway of last resort is 192.168.25.25 to network 0.0.0.0

192.168.25.0/29 is subnetted, 1 subnets

C 192.168.25.24 is directly connected, BRI0

141.130.0.0/32 is subnetted, 1 subnets

S 141.130.174.149 [2/0] via 172.31.26.1

172.31.0.0/24 is subnetted, 1 subnets

C 172.31.26.0 is directly connected, Ethernet0

10.0.0.0/32 is subnetted, 2 subnets

S 10.1.16.7 [1/0] via 172.31.26.2

S 10.11.1.63 [1/0] via 172.31.26.2

S* 0.0.0.0/0 [200/0] via 192.168.25.25

-----

Main router "show IP route"

Gateway of last resort is 172.31.22.153 to network 0.0.0.0 (<-- 172.31.22.153 is other end of the VPN connection here in NY. Problematic routers are in California)

4.0.0.0/32 is subnetted, 3 subnets

S 4.4.4.1 [254/0] via 12.119.241.81

S 4.4.4.2 [254/0] via 12.119.241.81

C 4.4.4.18 is directly connected, Loopback0

D EX 141.130.0.0/16 [170/15021568] via 172.31.22.153, 00:09:22, Tunnel0

172.31.0.0/16 is variably subnetted, 3 subnets, 2 masks

C 172.31.22.156/30 is directly connected, Tunnel1

C 172.31.22.152/30 is directly connected, Tunnel0

C 172.31.26.0/24 is directly connected, FastEthernet0/0

D EX 10.0.0.0/8 [170/15021568] via 172.31.22.153, 00:09:22, Tunnel0

12.0.0.0/8 is variably subnetted, 4 subnets, 2 masks

S 12.146.98.133/32 [1/0] via 12.119.241.81

S 12.146.98.130/32 [1/0] via 12.119.241.81

C 12.119.241.80/30 is directly connected, Serial0/1/0

C 12.119.241.81/32 is directly connected, Serial0/1/0

D*EX 0.0.0.0/0 [170/14509056] via 172.31.22.153, 00:12:17, Tunnel0

D EX 172.16.0.0/12 [170/15535872] via 172.31.22.153, 00:12:17, Tunnel0

-----

ISDN route "show dialer"

BRI0 - dialer type = ISDN

Dial String Successes Failures Last DNIS Last status

16319525672 22 876 00:42:47 failed

16319528981 18805 898 00:43:17 failed

0 incoming call(s) have been screened.

0 incoming call(s) rejected for callback.

BRI0:1 - dialer type = ISDN

Idle timer (120 secs), Fast idle timer (20 secs)

Wait for carrier (30 secs), Re-enable (15 secs)

Dialer state is idle

BRI0:2 - dialer type = ISDN

Idle timer (120 secs), Fast idle timer (20 secs)

Wait for carrier (30 secs), Re-enable (15 secs)

Dialer state is idle

-----

ISDN Router "show standby"

Local state is Standby, priority 100

Hellotime 3 sec, holdtime 10 sec

Next hello sent in 0.710

Virtual IP address is 172.31.26.1 configured

Active router is 172.31.26.2, priority 200 expires in 7.588

Standby router is local

1 state changes, last state change 8w1d

IP redundancy name is "hsrp-Et0-0" (default)

-----

Main Router "show standby"

FastEthernet0/0 - Group 0

State is Active

1 state change, last state change 29w6d

Virtual IP address is 172.31.26.1

Active virtual MAC address is 0000.0c07.ac00

Local virtual MAC address is 0000.0c07.ac00 (v1 default)

Hello time 3 sec, hold time 10 sec

Next hello sent in 0.312 secs

Preemption enabled

Active router is local

Standby router is 172.31.26.3, priority 100 (expires in 7.536 sec)

Priority 200 (configured 200)

IP redundancy name is "hsrp-Fa0/0-0" (default)

[Richard Burts]

Pete

Thanks for posting the additional information. I think that the most helpful part of it was the show standby. According to the ISDN router the last state change was

1 state changes, last state change 8w1d

and the main router says that the last state change was

1 state change, last state change 29w6d

I wonder about the difference (8 weeks vs 29 weeks) and wonder if perhaps the difference is because the ISDN was rebooted 8 weeks 1 day ago. You could check that with show version which includes the information about when the last reboot was.

But in any case if it has been at least 8 weeks since there was a state change then it is probably not a problem with HSRP that casued the high ISDN usage.

The show dialer would be especially useful if you could catch the ISDN router with an active call. But the output that you posted may have some helpful information. Especially these two lines:

16319525672 22 876 00:42:47 failed

16319528981 18805 898 00:43:17 failed

which show that 43 minutes ago the router attempted a call on the first number which failed and 30 seconds later attempted a call on the second number which also failed. This is the same time interval that you say the calls lasted. It looks to me like the router is attempting a call for some reason, the call fails and it attempts the second number which also fails.

So I think that there now are two questions to investigate: 1) what is causing the calls and 2) why are the calls failing.

Assuming that the end stations have the HSRP address configured as their default gateway, and assuming that HSRP has not made the ISDN router active recently I believe we can say that the cause of ISDN activity is not end user traffic. (and the discussion of static routes while interesting is probably not part of the problem) I believe that something on the router is generating traffic. It might very well be SNMP or it could be syslog, or it might be something else. But I believe it is generated on the ISDN router rather than being forwarded by the ISDN router.

It might be helpful if you could run some debug on the ISDN router. debug dialer might help answer the question of what is causing the dial activity. debug isdn q931 or debug ppp authentication or debug ppp negotiation might help answer the question of why the call is failing.

If you enable logging buffered you could let the debug output go to the logging buffer and not need to see the debug output in real time. You would use the show log command to see the content of the buffer.

HTH

Rick

[priedman1]

Hi Rick,

THanks again for the help. I'll try to debug the router but unfortunately, it looks like someone lost the enable secret password so I can't get into enable mode without doing a password recovery.

Pete

[priedman1]

Hi Rick,

Thanks again for all of your assistance. Here is some output from the debug I turned on. Hope it provides some insight.

R_US_LIV_BU#isdn test call interface bri 0 16319528981

R_US_LIV_BU#

8w6d: %SYS-2-LINKED: Bad enqueue of 2A64048 in queue 2A63E20

-Process= "ISDN", ipl= 0, pid= 39

-Traceback= 1C67B0 568554 57BE78 5722D0 56B06C 580648 55B498 55B468isdn test cal

l interface bri 0 16319528981

R_US_LIV_BU#term mon

8w6d: %SYS-2-LINKED: Bad enqueue of 2A65B28 in queue 2A646D0

-Process= "ISDN", ipl= 0, pid= 39

-Traceback= 1C67B0 568554 57BE78 5722D0 56B06C 580648 55B49sh isdn active

8w6d: ISDN BR0: Event: L2 Discard: L2 Session IDs global=188 pak=187

8w6d: ISDN BR0: Event: L2 Discard: L2 Session IDs global=198 pak=197

[Richard Burts]

Pete

I am not sure about the bad enqueue message but traceback messages are almost always a sign of a software problem. I would suggest that you check the version of software that you are running for any known bugs relating to ISDN.

It also would be interesting to see if you get the same errors if you test by pinging the address 192.168.25.25 that you get when you test with the isdn test call command.

HTH

Rick

[priedman1]

Hi Rick,

I managed to get some official Cisco support on this issue and it turns out that there is a bug in the version of IOS we are running on this IDSN router (12.2(29). The bug generates "tracebacks" and sends them to our syslog servers (overseas). This is what resulted in all of the 30 second calls on our bill.

I sent the debugs to Cisco and their investigation found this bug. Interesting how this just started happening with all of the ISDN calling. I would have figured that it is a hardware failure of sorts but they are saying software.

To alleviate all of the calling in the meantime, we removed logging from the config and Cisco also sent me a recommended access list that blocked additional traffic that may be deemed "interesting".

Thanks again for your help.

Regards,

Pete

[Richard Burts]

Pete

This is an interesting resolution of your problem. When I saw the traceback in one of the posts I assumed that there was a software problem. I am puzzled that the software was not exhibiting this behavior the month before. But I can easily believe that being more restrictive about what is interesting traffic is a good workaround for the issue.

The nice thing about restricting logging from being interesting traffic is that it will prevent logging from bringing up the ISDN but that if the ISDN is already up the logging messages will be transmitted.

I am glad that you have a resolution of your issue.

HTH

Rick

[Richard Burts]

Pete

There are several things that may be involved in finding out why there is so much ISDN usage. It may involve the identification of "interesting" traffic by the dialer group and dialer list. Or it may involve routing logic and what is sending traffic through the ISDN. It might also involve things like routing protocols which might attempt to send hellos or updates over the ISDN.

It would be very helpful if you post the config of both routers.

HTH

Rick