Slow ARP response for dial-in clients

[arrowsupport]

I’ve been experiencing an intermittent issue with remote PC’s connecting to a Cisco AS5350 Universal Gateway - basically, a RAS server.

The issue as far as I’ve been able to pinpoint seems to be related to the amount of time it takes the dial-in client to register an ARP entry on the local network where the RAS server and other servers are connected.   If I start an extend ping to one of the servers on the local network (not to the RAS server) once my dial-up connection has been established, I typically see anywhere between 3 and 18 ICMP request timeouts before I start receiving replies.  And if at the same time I start an extended ping to the IP address of the RAS server, ICMP replies are received immediately with no request timeouts.

Topology:

Dial-in Client <===> AS5350 RAS <===> L2 Switch <===> Server

192.168.240.131         240.5                           240.1               240.21

The switch that the AS5350 and the servers are connected to is a WS-C2960G-8TC-L layer-2 switch with a very basic config.  Basically they only thing I’ve changed during the course of my troubleshooting is the STP mode, STP forward time and to enabled STP portfast on the uplinks to the AS5350 and the server… see configuration below:

Current configuration : 2721 bytes

!

version 12.2

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service sequence-numbers

!

hostname Switch

!

boot-start-marker

boot-end-marker

!

no aaa new-model

system mtu routing 1500

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

spanning-tree vlan 1 forward-time 5

!

vlan internal allocation policy ascending

!

interface GigabitEthernet0/1

description Uplink to Server

spanning-tree portfast

!

interface GigabitEthernet0/2

description Uplink to CLE-AS5350 RAS

speed 100

duplex full

spanning-tree portfast

!

interface GigabitEthernet0/3

!

interface GigabitEthernet0/4

!

interface GigabitEthernet0/5

!

interface GigabitEthernet0/6

!

interface GigabitEthernet0/7

!

interface GigabitEthernet0/8

!

interface Vlan1

ip address 192.168.240.1 255.255.255.0

!

ip http server

ip http secure-server

!

line con 0

exec-timeout 0 0

logging synchronous

line vty 0 4

login

line vty 5 15

login

!

end

For troubleshooting, I enabled “debug arp” on the switch and attempted a dial-up connection to the AS5350.  Once the call was established and I received a DHCP lease (192.168.240.131), I started an extended ping to a server (192.168.240.21) on the network… see below:

Host Details:

192.168.240.1 (b4e9.b006.9e40) = Vlan1 on L2 switch.

192.168.240.21 (5cf9.dd48.76dd) = Server.

192.168.240.5 (000d.280c.fe1b) = Cisco AS5350 RAS server.

192.168.240.131 (0000.0000.0000) = PPP dial-in client on RAS server.

000292: *Mar  1 00:21:22.819 UTC: IP ARP: creating incomplete entry for IP address: 192.168.240.131 interface Vlan1

000293: *Mar  1 00:21:22.819 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000298: *Mar  1 00:21:27.013 UTC: IP ARP: rcvd req src 192.168.240.21 5cf9.dd48.76dd, dst 192.168.240.131 Vlan1

000299: *Mar  1 00:21:27.441 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000306: *Mar  1 00:21:32.441 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000314: *Mar  1 00:21:37.449 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000323: *Mar  1 00:21:42.440 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000329: *Mar  1 00:21:47.440 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000334: *Mar  1 00:21:52.439 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000344: *Mar  1 00:21:57.447 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000350: *Mar  1 00:22:02.447 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000358: *Mar  1 00:22:07.430 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000364: *Mar  1 00:22:12.438 UTC: IP ARP: creating incomplete entry for IP address: 192.168.240.131 interface Vlan1

000365: *Mar  1 00:22:12.438 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40,dst 192.168.240.131 0000.0000.0000 Vlan1

000372: *Mar  1 00:22:17.437 UTC: IP ARP: sent req src 192.168.240.1 b4e9.b006.9e40, dst 192.168.240.131 0000.0000.0000 Vlan1

000373: *Mar  1 00:22:17.446 UTC: IP ARP: rcvd rep src 192.168.240.131 000d.280c.fe1b, dst 192.168.240.1 Vlan1

The first line of the debug shows the switch creating an “incomplete entry” for the dial-in client (192.168.240.131).

For all subsequent ICMP requests, you can see that the dial-in client has a MAC address of 0000.0000.0000 – I guess you would call this an incomplete entry.

On the last line of the debug output, you can see that the dial-in client (192.168.240.131) finally gets the MAC address of the AS5350 (000d.280c.fe1b) assigned to it – this is when we start getting ICMP replies.

So during this capture, there were 12 ICMP request timeouts before the dial-in client started receiving replies.

Below is the current config on my Cisco AS5350 RAS server:

Current configuration : 6741 bytes

!

version 12.3

service timestamps debug datetime localtime show-timezone

service timestamps log datetime localtime show-timezone

service password-encryption

!

hostname AS5350

!

boot-start-marker

no boot startup-test

boot-end-marker

!

logging buffered 2048000 debugging

enable secret 5 *********************

!

resource-pool disable

calltracker enable

spe country usa

!

spe call-record modem

!

spe default-firmware spe-firmware-1

aaa new-model

!

aaa authentication login default group tacacs+ local

aaa authentication login NO_AUTHEN none

aaa authentication enable default group tacacs+ enable

aaa authentication ppp dialin if-needed local

aaa authorization exec default group tacacs+ local

aaa authorization commands 0 default group tacacs+ local none

aaa authorization commands 1 default group tacacs+ local none

aaa authorization commands 15 default group tacacs+ local none

aaa accounting exec default start-stop group tacacs+

aaa accounting commands 0 default start-stop group tacacs+

aaa accounting commands 1 default start-stop group tacacs+

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting network default start-stop group tacacs+

aaa session-id common

ip subnet-zero

!

!

ip cef

ip dhcp excluded-address 192.168.240.1 192.168.240.127

ip dhcp excluded-address 192.168.240.150 192.168.240.254

!

ip dhcp pool LOCAL

   network 192.168.240.0 255.255.255.0

   default-router 192.168.240.1

   lease 0 1

!

ip ssh time-out 10

ip ssh version 2

isdn switch-type primary-4ess

!

fax interface-type fax-mail

!

controller T1 3/0

shutdown

!

controller T1 3/1

framing esf

linecode b8zs

pri-group timeslots 1-24

description PRI on Copper

!

no crypto isakmp ccm

!

interface FastEthernet0/0

no ip address

shutdown

!

interface FastEthernet0/1

description Uplink to Switch – Gi0/2

ip address 192.168.240.5 255.255.255.0

duplex full

speed 100

!

interface Serial0/0

no ip address

shutdown

!

interface Serial0/1

no ip address

shutdown

!

interface Serial3/0:23

no ip address

shutdown

!

interface Serial3/1:23

description PRI on Copper

no ip address

encapsulation ppp

dialer rotary-group 2

dialer-group 2

isdn switch-type primary-4ess

isdn incoming-voice modem

isdn T306 60000

fair-queue

no cdp enable

!

interface Dialer2

ip unnumbered FastEthernet0/1

encapsulation ppp

dialer in-band

dialer idle-timeout 0

dialer-group 2

peer default ip address dhcp-pool LOCAL

fair-queue

no cdp enable

ppp authentication chap pap callin

ppp multilink

!

interface Group-Async0

no ip address

no group-range

!

interface Group-Async1

description Dial-up PRI modem lines

ip unnumbered FastEthernet0/1

encapsulation ppp

dialer in-band

dialer idle-timeout 0

async mode interactive

peer default ip address dhcp-pool LOCAL

fair-queue

ppp authentication chap pap callin

group-range 1/00 1/59

!

router eigrp 100

network 192.168.240.0

auto-summary

!

ip classless

ip route 0.0.0.0 0.0.0.0 192.168.240.1

ip tacacs source-interface FastEthernet0/1

no ip http server

no ip http secure-server

!

logging history debugging

logging trap debugging

logging x.x.x.x

access-list 101 deny   eigrp any any

access-list 101 permit ip any any

access-list 101 remark dialer-list used for dialer-list 1

access-list 182 remark *** PERMIT SSH TO THIS DEVICE ***

access-list 182 permit tcp any any eq 22

access-list 182 deny   ip  any any log

dialer-list 1 protocol ip  permit

!

tacacs-server host x.x.x.x

tacacs-server host x.x.x.x

tacacs-server directed-request

tacacs-server key 7 *******************

!

control-plane

!

voice-port 3/0:D

voice-port 3/1:D

!

dial-peer cor custom

!

ss7 mtp2-variant Bellcore 0

ss7 mtp2-variant Bellcore 1

ss7 mtp2-variant Bellcore 2

ss7 mtp2-variant Bellcore 3

!

line con 0

exec-timeout 0 0

logging synchronous

line aux 0

no exec

line vty 0 4

access-class 182 in

exec-timeout 30 0

logging synchronous

transport input ssh

escape-character BREAK

line 1/00 1/59

no modem callout

modem Dialin

rotary 1

transport input all

transport output all

autoselect during-login

autoselect ppp

!

scheduler allocate 10000 400

ntp clock-period 17180055

ntp server x.x.x.x

end

Cisco AS5350 IOS:  c5350-ik9s-mz.123-11.T11.bin

Is anyone aware of an IOS bug or an error in my configurations that could be causing the delay in creating an ARP entry for the dial-in client?

I am open to any suggestions.

BTW, if I add static arp entries on the server, ICMP replies are typically received after one or two request timeouts.

However, I feel this is not a solution to the problem, only a band-aid fix.

arp -s 192.168.240.128 00-0d-28-0c-fe-1b

arp -s 192.168.240.129 00-0d-28-0c-fe-1b

arp -s 192.168.240.130 00-0d-28-0c-fe-1b

arp -s 192.168.240.131 00-0d-28-0c-fe-1b

arp -s 192.168.240.132 00-0d-28-0c-fe-1b

arp -s 192.168.240.133 00-0d-28-0c-fe-1b

arp -s 192.168.240.134 00-0d-28-0c-fe-1b

arp -s 192.168.240.135 00-0d-28-0c-fe-1b

arp -s 192.168.240.136 00-0d-28-0c-fe-1b

arp -s 192.168.240.137 00-0d-28-0c-fe-1b

arp -s 192.168.240.138 00-0d-28-0c-fe-1b

arp -s 192.168.240.139 00-0d-28-0c-fe-1b

arp -s 192.168.240.140 00-0d-28-0c-fe-1b

arp -s 192.168.240.141 00-0d-28-0c-fe-1b

arp -s 192.168.240.142 00-0d-28-0c-fe-1b

arp -s 192.168.240.143 00-0d-28-0c-fe-1b

arp -s 192.168.240.144 00-0d-28-0c-fe-1b

arp -s 192.168.240.145 00-0d-28-0c-fe-1b

arp -s 192.168.240.146 00-0d-28-0c-fe-1b

arp -s 192.168.240.147 00-0d-28-0c-fe-1b

arp -s 192.168.240.148 00-0d-28-0c-fe-1b

arp -s 192.168.240.149 00-0d-28-0c-fe-1b

Thank you for taking the time to read my post.

-Brad